Deep Multi-Scale Video Prediction

ABSTRACT

In one embodiment, a method includes receiving a plurality of input frames of a video sequence associated with a time t, predicting, using a convolutional network, one or more future frames of the video sequence from the plurality of input frames, wherein the convolutional network is trained with randomly selected temporal sequences of a n×m grid of pixels from the plurality of input frames exhibiting a threshold of optical flow. In addition, the training may comprise randomly selecting temporal sequences of a n×m grid of pixels from the plurality of input frames exhibiting a threshold of optical flow.

PRIORITY

This application is a continuation under 35 U.S.C. § 120 of U.S. patentapplication Ser. No. 15/814,782, filed 16 Nov. 2017, which claims thebenefit, under 35 U.S.C. § 119(e), of U.S. Provisional PatentApplication No. 62/423,176, filed 16 Nov. 2016, each of which isincorporated herein by reference.

TECHNICAL FIELD

This disclosure generally relates to predicting future images from avideo sequence.

BACKGROUND

Predicting future images of a video sequence may require the use oftraining models that are able to accurately predict and construct imageevolution accurately. These training models may require the input of oneor more frames, and the prediction may be at a pixel-space level.

A social-networking system, which may include a social-networkingwebsite, may enable its users (such as persons or organizations) tointeract with it and with each other through it. The social-networkingsystem may, with input from a user, create and store in thesocial-networking system a user profile associated with the user. Theuser profile may include demographic information, communication-channelinformation, and information on personal interests of the user. Thesocial-networking system may also, with input from a user, create andstore a record of relationships of the user with other users of thesocial-networking system, as well as provide services (e.g., wall posts,photo-sharing, event organization, messaging, games, or advertisements)to facilitate social interaction between or among users.

The social-networking system may send over one or more networks contentor messages related to its services to a mobile or other computingdevice of a user. A user may also install software applications on amobile or other computing device of the user for accessing a userprofile of the user and other data within the social-networking system.The social-networking system may generate a personalized set of contentobjects to display to a user, such as a newsfeed of aggregated storiesof other users connected to the user.

A mobile computing device-such as a smartphone, tablet computer, orlaptop computer-may include functionality for determining its location,direction, or orientation, such as a GPS receiver, compass, gyroscope,or accelerometer. Such a device may also include functionality forwireless communication, such as BLUETOOTH communication, near-fieldcommunication (NFC), or infrared (IR) communication or communicationwith a wireless local area networks (WLANs) or cellular-telephonenetwork. Such a device may also include one or more cameras, scanners,touchscreens, microphones, or speakers. Mobile computing devices mayalso execute software applications, such as games, web browsers, orsocial-networking applications. With social-networking applications,users may connect, communicate, and share information with other usersin their social networks.

SUMMARY OF PARTICULAR EMBODIMENTS

In particular embodiments, a computing device may receive a plurality ofinput frames of a video sequence associated with a time t. Then, thecomputing device may train a convolutional network to predict one ormore future frames of the video sequence from the plurality of inputframes based on a generative model. The training may comprise using anadversarial model and an image gradient difference loss model. Thetraining may comprise randomly selecting temporal sequences of a n×mgrid of pixels from the plurality of input frames exhibiting a thresholdof optical flow. Thereafter, the computing device may output a firstfuture frame of the video sequence associated with a time t+1 aspredicted by the generative model. In particular embodiments, thecomputing device may predict a second future frame of the video sequenceassociated with time t+2 by using the first future frame as an inputframe of the video sequence, recursively apply the generative modelbased on the first future frame, and outputting the second future frameof the video sequence associated with the time t+2 as predicted by thegenerative model.

The embodiments disclosed above herein are only examples, and the scopeof this disclosure is not limited to them. Particular embodiments mayinclude all, some, or none of the components, elements, features,functions, operations, or steps of the embodiments disclosed above.Embodiments according to the invention are in particular disclosed inthe attached claims directed to a method, a storage medium, a system anda computer program product, wherein any feature mentioned in one claimcategory, e.g. method, can be claimed in another claim category, e.g.system, as well. The dependencies or references back in the attachedclaims are chosen for formal reasons only. However any subject matterresulting from a deliberate reference back to any previous claims (inparticular multiple dependencies) can be claimed as well, so that anycombination of claims and the features thereof are disclosed and can beclaimed regardless of the dependencies chosen in the attached claims.The subject-matter which can be claimed comprises not only thecombinations of features as set out in the attached claims but also anyother combination of features in the claims, wherein each featurementioned in the claims can be combined with any other feature orcombination of other features in the claims. Furthermore, any of theembodiments and features described or depicted herein can be claimed ina separate claim and/or in any combination with any embodiment orfeature described or depicted herein or with any of the features of theattached claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a next frame prediction convolution sequence.

FIG. 2 illustrates a multi-scale architecture for making predictions

FIGS. 3A and 3B illustrate the use of particular areas of video for usein future frame prediction.

FIGS. 4A-4F illustrate video clips from Sport1m for training videopredictions.

FIGS. 5A-5H illustrate video clips from UCF101 for training videopredictions.

FIGS. 6A-6D illustrate a generational results of 8×8 frames and 32×32patches.

FIG. 7 illustrates an example method for predicting future images from avideo sequence.

FIG. 8 illustrates an example network environment associated with asocial-networking system.

FIG. 9 illustrates an example social graph.

FIG. 10 illustrates an example computer system.

DESCRIPTION OF EXAMPLE EMBODIMENTS

In particular embodiments, learning to predict future images from avideo sequence may involve the construction of an internalrepresentation that models the image evolution accurately, in additionto its content and dynamics. This is why pixel-space video predictionmay be viewed as a promising avenue for unsupervised feature learning.In particular embodiments, many vision applications may benefit from theknowledge of upcoming frames of videos, for example, using a techniquethat does not require the complexity of tracking every pixel trajectory.Particular embodiments train a convolutional neural network (CNN) togenerate future frames given an input sequence. As an example and not byway of limitation, to deal with blurry predictions obtained from astandard Mean Squared Error (MSE) loss function, particular embodimentsmay utilize three different and complementary feature learningstrategies: a multi-scale architecture, an adversarial training method,and an image gradient difference loss function. Predictions generated byparticular embodiments are compared herein to different publishedresults based on recurrent neural network on the UCF101 dataset (e.g.,an action recognition data set of realistic action videos, collectedfrom YouTube, having 101 action categories).

Unsupervised feature learning of video representations is a promisingdirection of research because the resources are quasi-unlimited and theprogress remaining to achieve in this area are important. Particularembodiments disclosed herein address the problem of frame prediction. Asan example and not by way of limitation, a difference with the problemof image reconstruction is that the ability of a model to predict futureframes may require building accurate, nontrivial internalrepresentations, even in the absence of other constraints (such assparsity). The better the prediction of such system are, the better thefeature representations may be. In particular embodiments, learningrepresentations by predicting the next sequence of image features hasshown improved classification results on two action recognitiondatasets. Particular embodiments focus on predicting directly in pixelspace and trying to address the inherent problems related to thisapproach.

In particular embodiments, unsupervised learning may be achieved by useof a convolutional model that may be trained to predict sets of futurepossible actions, or by use of a convolutional network that may betrained to learn to linearize motion in the code space. Besidesunsupervised learning, a video predictive system may find applicationsin robotics, video compression, inpainting, other relevant applications,and any combination thereof.

Particular embodiments may address the problem of lack of sharpness inthe frame predictions by utilizing a new loss function based on theimage gradients, designed to preserve the sharpness of the frames.Particular embodiments includes different model architectures aredescribed herein: simple, multi-scale, adversarial. In addition,particular embodiments may utilize the gradient difference loss functiondescribed herein.

Models

Particular video prediction models will be described herein. Inparticular embodiments, let Y={Y¹, . . . , Y^(n)} be a sequence offrames to predict from input frames X={X¹, . . . , X^(m)} in a videosequence. FIG. 1 illustrates a next frame prediction convolutionsequence 100. As shown in FIG. 1, particular embodiments may utilize aCNN, alternating between convolutions and Rectified Linear Units (ReLU)from an input X 110, to a first feature map 112, to a second feature map114, to a third feature map 116, to a fourth feature map 118, to a fifthfeature map 120, to an output G(x). Such a network G, as displayed inFIG. 1, may be trained to predict one or several concatenated frames Yfrom the concatenated frames X by minimizing a distance, for instancel_(p) with p=1 or p=2, between the predicted frame and the true frame:

_(p)(X,Y)=l _(p)(G(X),Y)=∥G(X)−U∥ _(p) ^(p)   (Equation 1).

In particular embodiments, convolutions may only account for short-rangedependencies, limited by the size of their kernels. However, usingpooling would only be part of the solution since the output has to be ofthe same resolution as the input. In particular embodiments, there maybe a number of ways to avoid the loss of resolution brought about bypooling/subsampling while preserving long-range dependencies. As anexample and the not by way of limitation, one method is to have nopooling/subsampling but many convolution layers. As another example andnot by way of limitation, another method is to use connections that“skip” the pooling/unpooling pairs to preserve the high frequencyinformation. As yet another example and not by way of limitation,another method is to combine multiple scales linearly as in thereconstruction process of a Laplacian pyramid. Particular embodimentsmay use one or more of these techniques.

In particular embodiments, using an l₂ loss, and to a lesser extent l₁,may produce blurry predictions, which may become increasingly worse whenpredicting further in the future. If the probability distribution for anoutput pixel has two equally likely modes v₁ and v₂, the valuev_(avg)=(v₁+v₂)/2 may minimize the l₂ loss over the data, even ifv_(avg) has very low probability. In particular embodiments, in the caseof an l₁ norm, this effect may diminish but does not disappear as theoutput value would be the median of the set of equally likely values.Although this disclosure describes using convolution models in aparticular manner, this disclosure contemplates using convolution modelsin any suitable manner.

Multi-Scale Network

Particular embodiments may address the problem of convolutions onlyaccounting for short-range dependencies (limited by the size of theirkernels) by incorporating a multi-scale model. FIG. 2 illustrates amulti-scale architecture 200 for making predictions. In particularembodiments, a multi-scale version of the model may be defined asfollows: Let s₁, . . . , s_(N) scales be the sizes of the inputs of thenetwork. Particular embodiments may set s₁=4×4, s₂=8×8, s₃=16×16, ands₄=32×32. Let u_(k) be the upscaling operator toward size s_(k). LetX_(k) ^(i),Y_(k) ^(i) (denote the downscaled versions of X^(i) and Y^(i)of size s_(k), and G′_(k) be a network that learns to predictY_(k)−u_(k) (Y_(k−1)) from X_(k) and a coarse guess of Y_(k). Particularembodiments may recursively define the network G_(k), that makes aprediction Ŷ_(k) of size s_(k), by:

Ŷ _(k) =G _(k)(X)=u _(k)(Ŷ _(k−1))+G′ _(k)(X _(k) ,u _(k)(Ŷ _(k−1)))  (Equation 2).

In particular embodiments, using Equation 2, the network may make aseries of predictions, starting from the lowest resolution, and uses theprediction of size s_(k) as a starting point to make the prediction ofsize s_(k+1). At the lowest scale s₁, the network may take only X₁ as aninput. This architecture is illustrated in FIG. 2. As show in FIG. 2, atstep 210, X_(k/2) ¹, X_(k/2) ¹, . . . X_(k/2) ^(i) are input intonetwork G′_(k/2) at size k/2, which outputs a frame 220, which is inputat step 230, along with X_(k) ¹, X_(k) ², into network G′_(k) at size k.This results in output frame 240, which at step 250 is then combinedwith frame 220 to results in final output frame 260. In particularembodiments, the set of trainable parameters is denoted W_(G), and theminimization may be performed via Stochastic Gradient Descent (SGD).

In particular embodiments, despite the multi-scale architecture, thesearch of Y from X without making any assumption on the space ofpossible configurations may still lead to blurry predictions, because ofreliance upon using an ll₂ loss, and to a lesser extent ll₁. In order tofurther reduce this effect, particular embodiments may incorporate anadversarial strategy and the image gradient difference loss.

Adversarial Training

Particular embodiments based on generative adversarial networks maygenerate image patches from random noise using two networks trainedsimultaneously. Such networks may use a discriminative network D toestimate the probability that a sample comes from the dataset instead ofbeing produced by a generative model G. The two models aresimultaneously trained so that G learns to generate frames that are hardto classify by D, while D learns to discriminate the frames generated byG. In particular embodiments, when G is trained, it should not bepossible for D to perform better than chance.

Particular embodiments may adapt this approach for the purpose of frameprediction. The generative model G is typically the one described above.In particular embodiments, the discriminative model D takes a sequenceof frames and is trained to predict the probability that the last framesof the sequence are generated by G. In particular embodiments, only thelast frames may be either real or generated by G; the rest of thesequence may be from the dataset. This allows the discriminative modelto make use of temporal information, so that G may learn to producesequences that are temporally coherent with its input. In particularembodiments, since G may be conditioned on the input frames X, there isvariability in the input of the generator even in the absence of noise,so noise is not a necessity. Particular embodiments may train thenetwork with and without adding noise.

Particular embodiments use an adversarial loss function. An adversarialloss function may address issues caused by reliance upon using an l₂loss, and to a lesser extent l₁. In particular embodiments, for asequence of frames X=(X¹, . . . , X^(m)) in the dataset, the next framesmay be either Y=(Y¹, . . . , Y^(n)) or Y′=(Y′¹, . . . , Y′^(n)), withequal probability. In particular embodiments, training the network withan ll₂ loss may result in predicting the average framesY_(avg)=(Y+Y′)/2. However, the sequence (X, Y_(avg)), composed of theframes of X followed by the frames of Y_(avg), is not a likely sequence,and D may discriminate them easily. In particular embodiments, the onlysequences the model D may not be able to classify as fake are (X, Y) and(X, Y′).

In particular embodiments, the discriminative model D may be amulti-scale convolutional network with a single scalar output. Thetraining of the pair (G, D) consists of two alternated steps, describedbelow. Particular embodiments may use pure SGD (minibatches of size 1);alternative embodiments may generalize the algorithm to minibatches ofsize M by summing the losses over the samples.

When training discriminative model D, particular embodiments may utilizea sample (X, Y) from the dataset. In particular embodiments, X, andrespectively Y, is a sequence of m, and respectively n, frames.Particular embodiments train discriminative model D to classify theinput (X, Y) into class 1 and the input (X, G(X)) into class 0. As anexample and not by way of limitation, for each scale k, particularembodiments may perform one SGD iteration of D_(k) while keeping theweights of G fixed. In particular embodiments, discriminative model Dmay be trained within the target=1 for the datapoint (X_(k), Y_(k)), andthe target=0 for (X_(k), G_(k) (X_(k))). As an example and not by way oflimitation the loss function used to train D may be:

$\begin{matrix}{{\mathcal{L}_{adv}^{D}\left( {X,Y} \right)} = {{\sum\limits_{k = 1}^{N_{scales}}{L_{bce}\left( {{D_{k}\left( {X_{k},Y_{k}} \right)},1} \right)}} + {L_{bce}\left( {{D_{k}\left( {X_{k},{G_{k}(X)}} \right)},0} \right)}}} & \left( {{Equation}\mspace{14mu} 3} \right)\end{matrix}$

where L_(bce) is the binary cross-entropy loss, defined as

$\begin{matrix}{{L_{bce}\left( {Y,\hat{Y}} \right)} = {{- {\sum\limits_{i}{{\hat{Y}}_{i}{\log \left( Y_{i} \right)}}}} + {\left( {1 - {\hat{Y}}_{i}} \right){\log \left( {1 - Y_{i}} \right)}}}} & \left( {{Equation}\mspace{14mu} 4} \right)\end{matrix}$

where Y_(i) takes its values in {0, 1} and Ŷ_(i) in [0, 1].

When training generative model G, particular embodiments may utilize adifferent sample (X, Y) from the dataset. While keeping the weights of Dfixed, particular embodiments perform one SGD step on G to minimize theadversarial loss:

$\begin{matrix}{{\mathcal{L}_{adv}^{G}\left( {X,Y} \right)} = {\sum\limits_{k = 1}^{N_{scales}}{{L_{bce}\left( {{D_{k}\left( {X_{k},{G_{k}\left( X_{k} \right)}} \right)},1} \right)}.}}} & \left( {{Equation}\mspace{14mu} 5} \right)\end{matrix}$

In particular embodiments, minimizing this loss may mean that thegenerative model G is making the discriminative model D as “confused” aspossible, in the sense that D will not discriminate the predictioncorrectly. However, in practice, minimizing this loss alone may lead toinstability. G may generate samples that “confuse” D, without beingclose to Y. In turn, D may learn to discriminate these samples, leadingG to generate other “confusing” samples, and so on. To address this,particular embodiments may train the generator with a combined losscomposed of the adversarial loss and the L_(p) loss. The generator G maytherefore be trained to minimize λ_(adv)L_(adv)^(G)+λ_(lp)L_(p)λ_(adv)L_(adv)+λ_(lp) L_(p). In particular embodiments,there is a tradeoff to adjust, by the means of the λ_(adv)λ_(adv) andλ_(lp)λ_(lp) parameters, between sharp predictions due to theadversarial principle, and similarity with the ground truth brought bythe second term. This process is summarized in Algorithm 1, withminibatches of size M, described below.

Algorithm 1: Training adversarial networks for next frame generation Setthe learning rates ρ_(D) and ρ_(G), and weights λ_(adv), λ_(l) _(p) .while not converged do | Update the discriminator D: | Get M datasamples (X, Y) = (X⁽¹⁾, Y⁽¹⁾), . . . , (X^((M)), Y^((M))) |$W_{D} = {W_{D} - {\rho_{D}{\sum\limits_{i = 1}^{M}\; \frac{\partial{\mathcal{L}_{adv}^{D}\left( {X^{(i)},Y^{(i)}} \right)}}{\partial W_{D}}}}}$| Update the generator G: | Get M new data samples (X, Y) = (X⁽¹⁾,Y⁽¹⁾), . . . , (X^((M)), Y^((M))) └$W_{G} = {W_{G} - {\rho_{G}{\sum\limits_{i = 1}^{M}\left( {{\lambda_{adv}\frac{\partial{\mathcal{L}_{adv}^{G}\left( {X^{(i)},Y^{(i)}} \right)}}{\partial W_{G}}} + {\lambda_{l_{p}}\frac{\partial{\lambda_{l_{p}}\left( {X^{(i)},Y^{(i)}} \right)}}{\partial W_{G}}}} \right)}}}$

Image Gradient Difference Loss (GDL)

Another strategy to sharpen the image prediction is to directly penalizethe differences of image gradient predictions in the generative lossfunction. Particular embodiments define a new loss function, theGradient Difference Loss (GDL), that can be combined with a l_(p) and/oradversarial loss function. The GDL function between the ground truthimage Y, and the prediction G(X)=Ŷ may be given by

$\begin{matrix}{{{\mathcal{L}_{gdl}\left( {X,Y} \right)} = {{L_{gdl}\left( {\hat{Y},Y} \right)} = {{\sum\limits_{i,j}{{{{Y_{i,j} - Y_{{i - 1},j}}} - {{{\hat{Y}}_{i,j} - {\hat{Y}}_{{i - 1},j}}}}}^{\alpha}} + {{{{Y_{i,{j - 1}} - Y_{i,j}}} - {{{\hat{Y}}_{i,{j - 1}} - {\hat{Y}}_{i,j}}}}}^{\alpha}}}},} & \left( {{Equation}\mspace{14mu} 6} \right)\end{matrix}$

where α is an integer greater or equal to 1, and | |denotes the absolutevalue function. While a total variation regularization approach takesonly the reconstructed frame in input, in particular embodiments, anapproach may be taken in which the loss penalizes gradient differencesbetween the prediction and the true output. Particular embodimentsselect the simplest possible image gradient by considering the neighborpixel intensities differences, rather than adopting a more sophisticatednorm on a larger neighborhood, in order to keep the training time low.

Combining Losses

Particular embodiments combine the losses previously defined withdifferent weights. The final loss equation is:

=λ_(adv)

_(adv) ^(G)(X,Y)+λ_(l) _(p)

_(p)(X,Y)+λ_(gdl)

_(gdl)(x,Y)   (Equation 7).

Experimental Results (Using 4 Input Frames to Predict 1 Frame)

Discussed below is a quantitative evaluation of the quality of videopredictions performed by embodiments described herein. FIGS. 3A and 3Billustrate the use of particular areas of video for use in future frameprediction. FIGS. 4A-4F illustrate video clips from Sport1m for trainingvideo predictions. FIGS. 5A-5H illustrate video clips from UCF101 fortraining video predictions. In particular embodiments, twoconfigurations were trained and compared: (1) Use of 4 input frames topredict one future frame. In order to generate frames further in thefuture, the model may be applied recursively by using the newlygenerated frame as an input. (2) Use of 8 input frames to produce 8frames simultaneously, which represents a more difficult problem.

In particular embodiments, the Sports1m video clips shown in FIGS. 4A-4Fwere utilized for training because many of the frames in the UCF101video clips only have a very small portion of the image that is actuallymoving, while the rest of the image is just a fixed background. Inparticular embodiments, a network may be trained by randomly selectingtemporal sequences of patches of 32×32 pixels after making sure theyshow enough movement (quantified by the l₂ difference between theframes). The data patches are first normalized so that their valuesrange between −1 and 1.

In particular embodiments, multi-scale architectures may employed ingenerating the results discussed herein. The baseline models may use l₁and l₂ losses. The GDL-l₁ (respectively GDL-ll₂) model may use acombination of the GDL with α=1 (respectively α=2) and p=1 (respectivelyp=2) loss; the relative weights λ_(gdl)λ_(gdl) and λ_(lp)λ_(lp) areboth 1. The adversarial (Adv) model uses the adversarial loss, with p=2weighted by λ_(adv)λ_(adv)=0.05 and λ_(lp)λ_(lp)=1. Finally, the Adv+GDLmodel is a combination of the adversarial loss and the GDL, with thesame parameters as for Adv with αα=1 and λ_(gdl)λ_(gdl)=1.

Generative model G architecture is presented in Table 1 below.

TABLE 1 Network architecture (Input: 4 frames-output: 1 frame)Generative network scales G₁ G₂ G₃ G₄ Number of feature maps 128, 256,128 128, 256, 128 128, 256, 512, 256, 128 128, 256, 512, 256, 128 Conv.kernel size 3, 3, 3, 3 5, 3, 3, 5 5, 3, 3, 3, 5 7, 5, 5, 5, 5, 7Adversarial network scales D₁ D₂ D₃ D₄ Number of feature maps 64 64,128, 128 128, 256, 256 128, 256, 512, 128 Conv. kernel size (no padding)3 3, 3, 3 5, 5, 5 7, 7, 5, 5 Fully connected 512, 256 1024, 512 1024,512 1024, 512

Particular embodiments contain padded convolutions interlaced with ReLUnon-linearities. A hyperbolic tangent (Tanh) may be added at the end ofthe model to ensure that the output values are between −1 and 1. Thelearning rate ρ_(G)ρ_(G) may start at 0.04 and is reduced over time to0.005. The minibatch size may be set to 4, or 8 in the case of theadversarial training, to take advantage of GPU hardware capabilities.Particular embodiments may train the network on small patches, and sinceit is fully convolutional, it may be applied on larger images at testtime.

In particular embodiments, for adversarial training, the discriminativemodel D, also presented in Table 1, may use standard non-paddedconvolutions followed by fully connected layers and ReLUnon-linearities. For the largest scale s₄, a 2×2 pooling may be addedafter the convolutions. The network may be trained by setting thelearning rate ρ_(D)ρ_(D) to 0.02.

To evaluate the quality of the image predictions resulting from thedifferent tested systems, particular embodiments compute the Peak Signalto Noise Ratio (PSNR) between the true frame Y (e.g., a ground truthframe) and the prediction frame Ŷ:

${{PSNR}\left( {Y,\hat{Y}} \right)} = {10\; \log_{10}\frac{\max_{\hat{Y}}^{2}}{\frac{1}{N}{\sum\limits_{i = 0}^{N}\left( {Y_{i} - {\hat{Y}}_{i}} \right)^{2}}}}$

(Equation 8) where max_(Ŷ) is the maximum possible value of the imageintensities. In particular embodiments, a Structural Similarity IndexMeasure (SSIM) may range between −1 and 1, a larger score meaning agreater similarity between the two images.

To measure the loss of sharpness between the true frame and theprediction, particular embodiments define the following sharpnessmeasure based on the difference of gradients between two images Y and Ŷ:

$\begin{matrix}{{{{Sharp}.\mspace{11mu} {diff}.\left( {Y,\hat{Y}} \right)} = {10\; \log_{10}\frac{\max_{\hat{Y}}^{2}}{\frac{1}{N}\begin{pmatrix}{\sum_{i}\sum_{j}} \\{{\left( {{\nabla_{i}Y} + {\nabla_{j}Y}} \right) - \left( {{\nabla_{i}\hat{Y}} + {\nabla_{j}\hat{Y}}} \right)}}\end{pmatrix}}}},{where}} & \left( {{Equation}\mspace{14mu} 9} \right) \\{\mspace{79mu} {{{\nabla_{i}Y} = {{{Y_{i,j} - Y_{{i - 1},j}}}\mspace{14mu} {and}}}\; \mspace{85mu} {{\nabla_{j}Y} = {{{Y_{i,j} - Y_{i,{j - 1}}}}.}}}} & \left( {{Equation}\mspace{14mu} 10} \right)\end{matrix}$

In particular embodiments, a larger score is better. As an example,these quantitative measures on 378 test videos from UCF101 (as shown inFIGS. 5A-5H) are given in Table 2:

TABLE 2 Comparison of the accuracy of the predictions on 10% of theUCF101 test images. The different models have been trained given 4frames to predict the next one. Similarity and sharpness measuresevaluated only in the areas of movement. Our best model has beenfine-tuned on UCF101 after the training on Sports1m. 1^(st) frameprediction scores 2^(nd) frame prediction scores Similarity SimilarityPSNR SSIM Sharpness PSNR SSIM Sharpness single sc. l₂ 26.5 0.84 24.722.4 0.82 24.2 l₂ 27.6 0.86 24.7 22.5 0.81 24.2 l₁ 28.7 0.88 24.8 23.80.83 24.3 GDL l₁ 29.4 0.90 25.0 24.9 0.84 24.4 GDL l₁* 29.9 0.90 25.026.4 0.87 24.5 Adv* 30.6 0.89 25.2 26.1 0.85 24.2 Adv + GDL* 31.5 0.9125.4 28.0 0.87 25.1 Adv + GDL fine-tuned * 32.0 0.92 25.4 28.9 0.89 25.0Last input 28.6 0.89 24.6 26.3 0.87 24.2 Optical flow 31.6 0.93 25.328.2 0.90 24.7 *models fine-tuned on patches of size 64 × 64.

In particular embodiments, the different models have been trained byusing four frames to predict the next one. Similarity and sharpnessmeasures may be evaluated only in the areas of movement. In particularembodiments, one model has been fine-tuned on UCF101 after the trainingon Sports1m.

In particular embodiments, since it is trivial to predict pixel valuesin static areas, especially on the UCF101 dataset where most of theimages are still, particular embodiments focused evaluation in themoving areas. As shown in FIGS. 3A and 3B, evaluation of the accuracy offuture frames prediction may only take the moving areas of the imagesinto account. FIG. 3A illustrates an example of frame predictions in anentire image with ground truth. FIG. 3B illustrates images masked withthresholded optical flow so as to isolate the moving areas.

Particular embodiments compute the different quality measures only inthe areas where the optical flow is higher than a fixed threshold. Usingthe EpicFlow method (Edge-Preserving Interpolation of Correspondencesfor Optical Flow), if at least one color channel is lower than 0.2(image color range between 0 and 1), particular embodiments reset thecorresponding pixel intensity of the output and ground truth to 0, andcompute similarity measures in the resulting masked images. Similarityand sharpness measures computed on the whole images are discussed withrespect to Tables 3-5 and FIGS. 6A-6D, discussed below.

The numbers in Table 2 may indicate that all strategies perform betterthan the ll₂ predictions in terms of PSNR, SSIM, and sharpness. Inparticular embodiments, the multi-scale model may bring someimprovement, but used with an l₂ norm, it may not outperform simpleframe copy in the moving areas. In particular embodiments, the l₁ modelmay improve the results, since it replaces the mean by the median valueof individual pixel predictions. In particular embodiments, the GDL andadversarial predictions may lead to further gains, and finally thecombination of the multi-scale, l₁ norm, GDL, and adversarial trainingachieves the best PSNR, SSIM, and Sharpness difference measures.

While particular embodiments shows that the l₂ norm may be a poor metricfor training predictive models, particular embodiments also show thatthe PSNR at test time may be the worst for models trained optimizing thel₂ norm, although the PSNR is based on the l₂ metric. Particularembodiments also extrapolate the pixels of the next frame by propagatingthe optical flow from the previous ones.

FIGS. 4A-4F illustrate results on three video clips from the Sport1mdataset, as movements are more visible in this dataset. As describedabove, four input frames may be used to predict one future frame. Inparticular embodiments, a second output frame may be computedrecursively. Specifically, FIGS. 4A and 4B show four input frames 402that are taken from video of two people bowling, an l₁ norm result 404,a GDL result 406, a ground truth result 408 (e.g., the observed andactual next two frames), an l₂ norm result 410, an adversarial result412, and a combined adversarial and GDL result 414. FIGS. 4C and 4D showfour input frames 416 that are taken from video of a game of pool, an l₁norm result 418, a GDL result 420, a ground truth result 422, an l₂ normresult 424, an adversarial result 426, and a combined adversarial andGDL result 428. FIGS. 4E and 4F show four input frames 430 that aretaken from video of a person on a unicycle, an l_(i) norm result 432, aGDL result 434, a ground truth result 436, an l₂ norm result 438, anadversarial result 440, and a combined adversarial and GDL result 442.

Also described herein are the results of comparing the PSNR and SSIMvalues (with respect to target images FIGS. 5A and 5E) for variousembodiments described herein (FIGS. 5B, 5D, 5F, 5H) with respect toresults of existing techniques (FIGS. 5C and 5G) on the basketball dunkand ice dancing clips from UCF101. Specifically, FIGS. 5A-5D show frames510 of the basketball dunk clip for prediction 520 using a constantoptical flow (with resulting PSNR=25.4 (18.9) and SSIM=0.88 (0.56)), incomparison with the Ranzato prediction result 530, in comparison withthe Adv+GDL+l₁ result 540. In addition, FIGS. 5E-5H show frames 550 ofthe ice dancing clip for prediction 560 using a constant optical flow(with resulting PSNR=24.7 (20.6) and SSIM=0.84 (0.72)), in comparisonwith the Ranzato prediction result 570, in comparison with the Adv+GDL+1result 580. To obtain grayscale images, particular embodiments make RGBpredictions and extract the Y channel of our Adv+GDL model. Two framepredictions are shown for each method along with a 2× zoom of eachimage. The PSNR and SSIM values are computed in the moving areas of theimages, which comprise more than the 2/3 of the pixels in these exampleframes. The values in parentheses correspond to the second frameprediction measures.

Experimental Results (8 Frames×8 Frames and 32×32 Patches)

Described herein is the result of training the different multi-scalemodels, architecture described in Table 3, with 8 input frames topredict 8 frames simultaneously.

TABLE 3 Network architecture Models 8 frames in input-8 frames in outputGenerative network scales G₁ G₂ G₃ G₄ Number of feature maps 16, 32, 6416, 32, 64 32, 64, 32, 64, 128 128, 128 Conv. kernel size 3, 3, 3, 3 5,3, 3, 3 5, 5, 5, 7, 5, 5, 5 5, 5 Adversarial network scales D₁ D₂ D₃ D₄Number of feature maps 16 16, 32, 32 32, 64, 32, 64, 64 128, 128 Cony.kernel size (no padding) 3 3, 3, 3 5, 5, 5 7, 7, 5, 5 Fully connected128, 64 256, 128 256, 128 256, 128

Image similarity measures are given between the ground truth and thepredictions in Table 4:

TABLE 4 Comparison of the accuracy of the predictions on 10% of theUCF101 test images. The different models have been trained given 8frames to predict the 8 next ones. 1^(st) frame prediction scores 8^(th)frame prediction scores Similarity Similarity PSNR SSIM Sharpness PSNRSSIM Sharpness l₂ 18.3 0.59 17.5 15.4 0.51 17.4 Adv 21.1 0.61 17.6 17.10.52 17.4 l₁ 21.3 0.66 17.7 17.0 0.55 17.5 GDL l₁ 21.4 0.69 17.9 17.70.58 17.5 Last input 30.6 0.90 22.3 21.0 0.74 18.5

As shown above, for the first and eighth predicted frames, the numbersindicate that all strategies perform better than the l₂ predictions interms of PSNR and sharpness. The l₁ model, by replacing the meanintensity by the median value in individual pixel predictions, mayimprove results. The adversarial predictions may lead to further gains,and finally the GDL may allow the predictions to achieve the best PNSRand sharpness. In particular embodiments, the size of the networkemployed in the simultaneous prediction configuration is smaller than inthe unique frame prediction setting.

FIGS. 6A and 6B together show a generation result (on a UCF101 video) ofeight frames simultaneously, using a large version of the GDL ll₁ modelin which all the number of feature maps were multiplied by four. The toprow 610 and 614 of images across both FIGS. 6A and 6B illustratesvarious target frames; the bottom row of images 612 and 616 across bothFIGS. 6A and 6B illustrates predictions generated using embodimentsdescribed herein. In particular embodiments, compared to recursive frameprediction, predicting several inputs simultaneously may lead to betterlong-term results but worse short-term ones. The gap between the twoperformances may be reduced by the design of time multi-scalestrategies.

FIGS. 6C and 6D show a comparison of different methods 618 and 620 topredict 32×32 patches from UCF101, with predictions based on LSTMs usingsequences of patches. In particular embodiments, the model rankingestablished on UCF101 in terms of sharpness and PSNR may remainunchanged on the two sequences. When employing the setting 8 inputs-8outputs described in Table 3, the LSTM first frame prediction may appearsharper than a predicted frame, however when looking at a longer termfuture, predicted frames based on the gradient difference loss may leadto sharper results. In particular embodiments, by comparing visually theGDL l₁ and GDL l₂, the predictions appear to have a “chessboard” effectin the l₂ case. On the other hand, when employing the recursive strategy(4 inputs, 1 output), the adversarial training may lead to much sharperpredictions. While it may not look like anything close to the groundtruth on the long term, but it may remain realistic.

Particular embodiments deal with the evaluation of the classificationperformances of the learned representations in a weakly supervisedcontext, for instance on the UCF101 dataset. Another extension of thiswork could be the combination of the current system with optical flowpredictions. Alternatively, particular embodiments may replace opticalflow predictions in applications that do not explicitly require opticalflow but rather next frame predictions (e.g., causal (where the nextframe is unknown) segmentation of video streams).

As discussed above, the model described in Table 1 was trained with thedifferent losses to predict one frame from the four previous ones. Table5 shows similarity (PSNR and SSIM) and sharpness measures between thedifferent tested models predictions and frame to predict. In particularembodiments, the evaluation may be performed on the full images but maynot be really meaningful because predicting the future location ofstatic pixels is most accurately done by copying the last input frame.

TABLE 5 Comparison of the accuracy of the predictions on 10% of theUCF101 test images. The different models have been trained given 4frames to predict the next one. Similarity and sharpness measures onfull images. 1^(st) frame prediction scores 2^(nd) frame predictionscores Similarity Similarity PSNR SSIM Sharpness PSNR SSIM Sharpnesssingle sc. l₂ 19.0 0.59 17.8 14.2 0.48 17.5 l₂ 20.1 0.64 17.8 14.1 0.5017.4 l₁ 22.3 0.74 18.5 16.0 0.56 17.6 GDL l₁ 23.9 0.80 18.7 18.6 0.6417.7 Adv 24.4 0.77 18.7 18.9 0.59 17.3 Adv + GDL 27.2 0.83 19.6 22.60.72 18.5 Adv + GDL fine-tuned 29.6 0.90 20.3 26.0 0.83 19.4 Last input30.0 0.90 22.1 25.8 0.84 20.3

Systems and Methods

FIG. 7 illustrates an example method 700 for predicting future imagesfrom a video sequence. The method may begin at step 710, where acomputing device may receive a plurality of input frames of a videosequence associated with a time t. At step 720, the computing device maytrain a convolutional network to predict one or more future frames ofthe video sequence from the plurality of input frames based on agenerative model. The training may comprise using an adversarial modeland an image gradient difference loss model. The training may compriserandomly selecting temporal sequences of a n×m grid of pixels from theplurality of input frames exhibiting a threshold of optical flow. Atstep 730, the computing device may output a first future frame of thevideo sequence associated with a time t+1 as predicted by the generativemodel. In particular embodiments, the computing device may predict asecond future frame of the video sequence associated with time t+2 byusing the first future frame as an input frame of the video sequence,recursively apply the generative model based on the first future frame,and outputting the second future frame of the video sequence associatedwith the time t+2 as predicted by the generative model. In particularembodiments, the threshold of optical flow may be determined based on acomparison of movement in the n×m grid of pixels between a time t−1 andthe time t. In particular embodiments, the comparison of movement maycomprise comparing pixel intensity associated with each pixel of the n×mgrid of pixels between the time t−1 and the time t and calculating asimilarity measure based on compared pixel intensities. In particularembodiments, the training of the convolutional network may furthercomprise evaluating a quality of the predicted one or more future framesof the video sequence by calculating a peak-signal-to-noise ratiobetween the one or more future frames of the video sequence and a groundtruth frame of the video sequence. In particular embodiments, thetraining of the convolutional network may further comprise evaluating aquality of the predicted one or more future frames of the video sequenceby calculating a structural similarity index measure between the one ormore future frames of the video sequence and a ground truth frame of thevideo sequence. Particular embodiments may repeat one or more steps ofthe method of FIG. 7, where appropriate. Although this disclosuredescribes and illustrates particular steps of the method of FIG. 7 asoccurring in a particular order, this disclosure contemplates anysuitable steps of the method of FIG. 7 occurring in any suitable order.Moreover, although this disclosure describes and illustrates an examplemethod for predicting future images from a video sequence including theparticular steps of the method of FIG. 7, this disclosure contemplatesany suitable method for predicting future images from a video sequenceincluding any suitable steps, which may include all, some, or none ofthe steps of the method of FIG. 7, where appropriate. Furthermore,although this disclosure describes and illustrates particularcomponents, devices, or systems carrying out particular steps of themethod of FIG. 7, this disclosure contemplates any suitable combinationof any suitable components, devices, or systems carrying out anysuitable steps of the method of FIG. 7.

FIG. 8 illustrates an example network environment 800 associated with asocial-networking system. Network environment 800 includes a clientsystem 830, a social-networking system 860, and a third-party system 870connected to each other by a network 810. Although FIG. 8 illustrates aparticular arrangement of client system 830, social-networking system860, third-party system 870, and network 810, this disclosurecontemplates any suitable arrangement of client system 830,social-networking system 860, third-party system 870, and network 810.As an example and not by way of limitation, two or more of client system830, social-networking system 860, and third-party system 870 may beconnected to each other directly, bypassing network 810. As anotherexample, two or more of client system 830, social-networking system 860,and third-party system 870 may be physically or logically co-locatedwith each other in whole or in part. Moreover, although FIG. 8illustrates a particular number of client systems 830, social-networkingsystems 860, third-party systems 870, and networks 810, this disclosurecontemplates any suitable number of client systems 830,social-networking systems 860, third-party systems 870, and networks810. As an example and not by way of limitation, network environment 800may include multiple client system 830, social-networking systems 860,third-party systems 870, and networks 810.

This disclosure contemplates any suitable network 810. As an example andnot by way of limitation, one or more portions of network 810 mayinclude an ad hoc network, an intranet, an extranet, a virtual privatenetwork (VPN), a local area network (LAN), a wireless LAN (WLAN), a widearea network (WAN), a wireless WAN (WWAN), a metropolitan area network(MAN), a portion of the Internet, a portion of the Public SwitchedTelephone Network (PSTN), a cellular telephone network, or a combinationof two or more of these. Network 810 may include one or more networks810.

Links 850 may connect client system 830, social-networking system 860,and third-party system 870 to communication network 810 or to eachother. This disclosure contemplates any suitable links 850. Inparticular embodiments, one or more links 850 include one or morewireline (such as for example Digital Subscriber Line (DSL) or Data OverCable Service Interface Specification (DOCSIS)), wireless (such as forexample Wi-Fi or Worldwide Interoperability for Microwave Access(WiMAX)), or optical (such as for example Synchronous Optical Network(SONET) or Synchronous Digital Hierarchy (SDH)) links. In particularembodiments, one or more links 850 each include an ad hoc network, anintranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, aportion of the Internet, a portion of the PSTN, a cellulartechnology-based network, a satellite communications technology-basednetwork, another link 850, or a combination of two or more such links850. Links 850 need not necessarily be the same throughout networkenvironment 800. One or more first links 850 may differ in one or morerespects from one or more second links 850.

In particular embodiments, client system 830 may be an electronic deviceincluding hardware, software, or embedded logic components or acombination of two or more such components and capable of carrying outthe appropriate functionalities implemented or supported by clientsystem 830. As an example and not by way of limitation, a client system830 may include a computer system such as a desktop computer, notebookor laptop computer, netbook, a tablet computer, e-book reader, GPSdevice, camera, personal digital assistant (PDA), handheld electronicdevice, cellular telephone, smartphone, augmented/virtual realitydevice, other suitable electronic device, or any suitable combinationthereof. This disclosure contemplates any suitable client systems 830. Aclient system 830 may enable a network user at client system 830 toaccess network 810. A client system 830 may enable its user tocommunicate with other users at other client systems 830.

In particular embodiments, client system 830 may include a web browser832, and may have one or more add-ons, plug-ins, or other extensions. Auser at client system 830 may enter a Uniform Resource Locator (URL) orother address directing the web browser 832 to a particular server (suchas server 862, or a server associated with a third-party system 870),and the web browser 832 may generate a Hyper Text Transfer Protocol(HTTP) request and communicate the HTTP request to server. The servermay accept the HTTP request and communicate to client system 830 one ormore Hyper Text Markup Language (HTML) files responsive to the HTTPrequest. Client system 830 may render a webpage based on the HTML filesfrom the server for presentation to the user. This disclosurecontemplates any suitable webpage files. As an example and not by way oflimitation, webpages may render from HTML files, Extensible Hyper TextMarkup Language (XHTML) files, or Extensible Markup Language (XML)files, according to particular needs. Such pages may also executescripts such as, for example and without limitation, those written inJAVASCRIPT, JAVA, MICROSOFT SILVERLIGHT, combinations of markup languageand scripts such as AJAX (Asynchronous JAVASCRIPT and XML), and thelike. Herein, reference to a webpage encompasses one or morecorresponding webpage files (which a browser may use to render thewebpage) and vice versa, where appropriate.

In particular embodiments, social-networking system 860 may be anetwork-addressable computing system that can host an online socialnetwork. Social-networking system 860 may generate, store, receive, andsend social-networking data, such as, for example, user-profile data,concept-profile data, social-graph information, or other suitable datarelated to the online social network. Social-networking system 860 maybe accessed by the other components of network environment 800 eitherdirectly or via network 810. As an example and not by way of limitation,client system 830 may access social-networking system 860 using a webbrowser 832, or a native application associated with social-networkingsystem 860 (e.g., a mobile social-networking application, a messagingapplication, another suitable application, or any combination thereof)either directly or via network 810. In particular embodiments,social-networking system 860 may include one or more servers 862. Eachserver 862 may be a unitary server or a distributed server spanningmultiple computers or multiple datacenters. Servers 862 may be ofvarious types, such as, for example and without limitation, web server,news server, mail server, message server, advertising server, fileserver, application server, exchange server, database server, proxyserver, another server suitable for performing functions or processesdescribed herein, or any combination thereof. In particular embodiments,each server 862 may include hardware, software, or embedded logiccomponents or a combination of two or more such components for carryingout the appropriate functionalities implemented or supported by server862. In particular embodiments, social-networking system 860 may includeone or more data stores 864. Data stores 864 may be used to storevarious types of information. In particular embodiments, the informationstored in data stores 864 may be organized according to specific datastructures. In particular embodiments, each data store 864 may be arelational, columnar, correlation, or other suitable database. Althoughthis disclosure describes or illustrates particular types of databases,this disclosure contemplates any suitable types of databases. Particularembodiments may provide interfaces that enable a client system 830, asocial-networking system 860, or a third-party system 870 to manage,retrieve, modify, add, or delete, the information stored in data store864.

In particular embodiments, social-networking system 860 may store one ormore social graphs in one or more data stores 864. In particularembodiments, a social graph may include multiple nodes—which may includemultiple user nodes (each corresponding to a particular user) ormultiple concept nodes (each corresponding to a particular concept)—andmultiple edges connecting the nodes. Social-networking system 860 mayprovide users of the online social network the ability to communicateand interact with other users. In particular embodiments, users may jointhe online social network via social-networking system 860 and then addconnections (e.g., relationships) to a number of other users ofsocial-networking system 860 to whom they want to be connected. Herein,the term “friend” may refer to any other user of social-networkingsystem 860 with whom a user has formed a connection, association, orrelationship via social-networking system 860.

In particular embodiments, social-networking system 860 may provideusers with the ability to take actions on various types of items orobjects, supported by social-networking system 860. As an example andnot by way of limitation, the items and objects may include groups orsocial networks to which users of social-networking system 860 maybelong, events or calendar entries in which a user might be interested,computer-based applications that a user may use, transactions that allowusers to buy or sell items via the service, interactions withadvertisements that a user may perform, or other suitable items orobjects. A user may interact with anything that is capable of beingrepresented in social-networking system 860 or by an external system ofthird-party system 870, which is separate from social-networking system860 and coupled to social-networking system 860 via a network 810.

In particular embodiments, social-networking system 860 may be capableof linking a variety of entities. As an example and not by way oflimitation, social-networking system 860 may enable users to interactwith each other as well as receive content from third-party systems 870or other entities, or to allow users to interact with these entitiesthrough an application programming interfaces (API) or othercommunication channels.

In particular embodiments, a third-party system 870 may include one ormore types of servers, one or more data stores, one or more interfaces,including but not limited to APIs, one or more web services, one or morecontent sources, one or more networks, or any other suitable components,e.g., that servers may communicate with. A third-party system 870 may beoperated by a different entity from an entity operatingsocial-networking system 860. In particular embodiments, however,social-networking system 860 and third-party systems 870 may operate inconjunction with each other to provide social-networking services tousers of social-networking system 860 or third-party systems 870. Inthis sense, social-networking system 860 may provide a platform, orbackbone, which other systems, such as third-party systems 870, may useto provide social-networking services and functionality to users acrossthe Internet.

In particular embodiments, a third-party system 870 may include athird-party content object provider. A third-party content objectprovider may include one or more sources of content objects, which maybe communicated to a client system 830. As an example and not by way oflimitation, content objects may include information regarding things oractivities of interest to the user, such as, for example, movie showtimes, movie reviews, restaurant reviews, restaurant menus, productinformation and reviews, or other suitable information. As anotherexample and not by way of limitation, content objects may includeincentive content objects, such as coupons, discount tickets, giftcertificates, or other suitable incentive objects.

In particular embodiments, social-networking system 860 also includesuser-generated content objects, which may enhance a user's interactionswith social-networking system 860. User-generated content may includeanything a user can add, upload, send, or “post” to social-networkingsystem 860. As an example and not by way of limitation, a usercommunicates posts to social-networking system 860 from a client system830. Posts may include data such as status updates or other textualdata, location information, photos, videos, links, music or othersimilar data or media. Content may also be added to social-networkingsystem 860 by a third-party through a “communication channel,” such as anewsfeed or stream.

In particular embodiments, social-networking system 860 may include avariety of servers, sub-systems, programs, modules, logs, and datastores. In particular embodiments, social-networking system 860 mayinclude one or more of the following: a web server, action logger,API-request server, relevance-and-ranking engine, content-objectclassifier, notification controller, action log,third-party-content-object-exposure log, inference module,authorization/privacy server, search module, advertisement-targetingmodule, user-interface module, user-profile store, connection store,third-party content store, or location store. Social-networking system860 may also include suitable components such as network interfaces,security mechanisms, load balancers, failover servers,management-and-network-operations consoles, other suitable components,or any suitable combination thereof. In particular embodiments,social-networking system 860 may include one or more user-profile storesfor storing user profiles. A user profile may include, for example,biographic information, demographic information, behavioral information,social information, or other types of descriptive information, such aswork experience, educational history, hobbies or preferences, interests,affinities, or location. Interest information may include interestsrelated to one or more categories. Categories may be general orspecific. As an example and not by way of limitation, if a user “likes”an article about a brand of shoes the category may be the brand, or thegeneral category of “shoes” or “clothing.” A connection store may beused for storing connection information about users. The connectioninformation may indicate users who have similar or common workexperience, group memberships, hobbies, educational history, or are inany way related or share common attributes. The connection informationmay also include user-defined connections between different users andcontent (both internal and external). A web server may be used forlinking social-networking system 860 to one or more client systems 830or one or more third-party system 870 via network 810. The web servermay include a mail server or other messaging functionality for receivingand routing messages between social-networking system 860 and one ormore client systems 830. An API-request server may allow a third-partysystem 870 to access information from social-networking system 860 bycalling one or more APIs. An action logger may be used to receivecommunications from a web server about a user's actions on or offsocial-networking system 860. In conjunction with the action log, athird-party-content-object log may be maintained of user exposures tothird-party-content objects. A notification controller may provideinformation regarding content objects to a client system 830.Information may be pushed to a client system 830 as notifications, orinformation may be pulled from client system 830 responsive to a requestreceived from client system 830. Authorization servers may be used toenforce one or more privacy settings of the users of social-networkingsystem 860. A privacy setting of a user determines how particularinformation associated with a user can be shared. The authorizationserver may allow users to opt in to or opt out of having their actionslogged by social-networking system 860 or shared with other systems(e.g., third-party system 870), such as, for example, by settingappropriate privacy settings. Third-party-content-object stores may beused to store content objects received from third parties, such as athird-party system 870. Location stores may be used for storing locationinformation received from client systems 830 associated with users.Advertisement-pricing modules may combine social information, thecurrent time, location information, or other suitable information toprovide relevant advertisements, in the form of notifications, to auser.

FIG. 9 illustrates example social graph 900. In particular embodiments,social-networking system 860 may store one or more social graphs 900 inone or more data stores. In particular embodiments, social graph 900 mayinclude multiple nodes-which may include multiple user nodes 902 ormultiple concept nodes 904—and multiple edges 906 connecting the nodes.Example social graph 900 illustrated in FIG. 9 is shown, for didacticpurposes, in a two-dimensional visual map representation. In particularembodiments, a social-networking system 860, client system 830, orthird-party system 870 may access social graph 900 and relatedsocial-graph information for suitable applications. The nodes and edgesof social graph 900 may be stored as data objects, for example, in adata store (such as a social-graph database). Such a data store mayinclude one or more searchable or queryable indexes of nodes or edges ofsocial graph 900.

In particular embodiments, a user node 902 may correspond to a user ofsocial-networking system 860. As an example and not by way oflimitation, a user may be an individual (human user), an entity (e.g.,an enterprise, business, or third-party application), or a group (e.g.,of individuals or entities) that interacts or communicates with or oversocial-networking system 860. In particular embodiments, when a userregisters for an account with social-networking system 860,social-networking system 860 may create a user node 902 corresponding tothe user, and store the user node 902 in one or more data stores. Usersand user nodes 902 described herein may, where appropriate, refer toregistered users and user nodes 902 associated with registered users. Inaddition or as an alternative, users and user nodes 902 described hereinmay, where appropriate, refer to users that have not registered withsocial-networking system 860. In particular embodiments, a user node 902may be associated with information provided by a user or informationgathered by various systems, including social-networking system 860. Asan example and not by way of limitation, a user may provide his or hername, profile picture, contact information, birth date, sex, maritalstatus, family status, employment, education background, preferences,interests, or other demographic information. In particular embodiments,a user node 902 may be associated with one or more data objectscorresponding to information associated with a user. In particularembodiments, a user node 902 may correspond to one or more webpages.

In particular embodiments, a concept node 904 may correspond to aconcept. As an example and not by way of limitation, a concept maycorrespond to a place (such as, for example, a movie theater,restaurant, landmark, or city); a website (such as, for example, awebsite associated with social-network system 860 or a third-partywebsite associated with a web-application server); an entity (such as,for example, a person, business, group, sports team, or celebrity); aresource (such as, for example, an audio file, video file, digitalphoto, text file, structured document, or application) which may belocated within social-networking system 860 or on an external server,such as a web-application server; real or intellectual property (suchas, for example, a sculpture, painting, movie, game, song, idea,photograph, or written work); a game; an activity; an idea or theory; anobject in a augmented/virtual reality environment; another suitableconcept; or two or more such concepts. A concept node 904 may beassociated with information of a concept provided by a user orinformation gathered by various systems, including social-networkingsystem 860. As an example and not by way of limitation, information of aconcept may include a name or a title; one or more images (e.g., animage of the cover page of a book); a location (e.g., an address or ageographical location); a website (which may be associated with a URL);contact information (e.g., a phone number or an email address); othersuitable concept information; or any suitable combination of suchinformation. In particular embodiments, a concept node 904 may beassociated with one or more data objects corresponding to informationassociated with concept node 904. In particular embodiments, a conceptnode 904 may correspond to one or more webpages.

In particular embodiments, a node in social graph 900 may represent orbe represented by a webpage (which may be referred to as a “profilepage”). Profile pages may be hosted by or accessible tosocial-networking system 860. Profile pages may also be hosted onthird-party websites associated with a third-party system 870. As anexample and not by way of limitation, a profile page corresponding to aparticular external webpage may be the particular external webpage andthe profile page may correspond to a particular concept node 904.Profile pages may be viewable by all or a selected subset of otherusers. As an example and not by way of limitation, a user node 902 mayhave a corresponding user-profile page in which the corresponding usermay add content, make declarations, or otherwise express himself orherself. As another example and not by way of limitation, a concept node904 may have a corresponding concept-profile page in which one or moreusers may add content, make declarations, or express themselves,particularly in relation to the concept corresponding to concept node904.

In particular embodiments, a concept node 904 may represent athird-party webpage or resource hosted by a third-party system 870. Thethird-party webpage or resource may include, among other elements,content, a selectable or other icon, or other inter-actable object(which may be implemented, for example, in JavaScript, AJAX, or PHPcodes) representing an action or activity. As an example and not by wayof limitation, a third-party webpage may include a selectable icon suchas “like,” “check-in,” “eat,” “recommend,” or another suitable action oractivity. A user viewing the third-party webpage may perform an actionby selecting one of the icons (e.g., “check-in”), causing a clientsystem 830 to send to social-networking system 860 a message indicatingthe user's action. In response to the message, social-networking system860 may create an edge (e.g., a check-in-type edge) between a user node902 corresponding to the user and a concept node 904 corresponding tothe third-party webpage or resource and store edge 906 in one or moredata stores.

In particular embodiments, a pair of nodes in social graph 900 may beconnected to each other by one or more edges 906. An edge 906 connectinga pair of nodes may represent a relationship between the pair of nodes.In particular embodiments, an edge 906 may include or represent one ormore data objects or attributes corresponding to the relationshipbetween a pair of nodes. As an example and not by way of limitation, afirst user may indicate that a second user is a “friend” of the firstuser. In response to this indication, social-networking system 860 maysend a “friend request” to the second user. If the second user confirmsthe “friend request,” social-networking system 860 may create an edge906 connecting the first user's user node 902 to the second user's usernode 902 in social graph 900 and store edge 906 as social-graphinformation in one or more of data stores 864. In the example of FIG. 9,social graph 900 includes an edge 906 indicating a friend relationbetween user nodes 902 of user “A” and user “B” and an edge indicating afriend relation between user nodes 902 of user “C” and user “B.”Although this disclosure describes or illustrates particular edges 906with particular attributes connecting particular user nodes 902, thisdisclosure contemplates any suitable edges 906 with any suitableattributes connecting user nodes 902. As an example and not by way oflimitation, an edge 906 may represent a friendship, family relationship,business or employment relationship, fan relationship (including, e.g.,liking, etc.), follower relationship, visitor relationship (including,e.g., accessing, viewing, checking-in, sharing, etc.), subscriberrelationship, superior/subordinate relationship, reciprocalrelationship, non-reciprocal relationship, another suitable type ofrelationship, or two or more such relationships. Moreover, although thisdisclosure generally describes nodes as being connected, this disclosurealso describes users or concepts as being connected. Herein, referencesto users or concepts being connected may, where appropriate, refer tothe nodes corresponding to those users or concepts being connected insocial graph 900 by one or more edges 906.

In particular embodiments, an edge 906 between a user node 902 and aconcept node 904 may represent a particular action or activity performedby a user associated with user node 902 toward a concept associated witha concept node 904. As an example and not by way of limitation, asillustrated in FIG. 9, a user may “like,” “attended,” “played,”“listened,” “cooked,” “worked at,” or “watched” a concept, each of whichmay correspond to an edge type or subtype. A concept-profile pagecorresponding to a concept node 904 may include, for example, aselectable “check in” icon (such as, for example, a clickable “check in”icon) or a selectable “add to favorites” icon. Similarly, after a userclicks these icons, social-networking system 860 may create a “favorite”edge or a “check in” edge in response to a user's action correspondingto a respective action. As another example and not by way of limitation,a user (user “C”) may listen to a particular song (“Imagine”) using aparticular application (a third-party online music application). In thiscase, social-networking system 860 may create a “listened” edge 906 anda “used” edge (as illustrated in FIG. 9) between user nodes 902corresponding to the user and concept nodes 904 corresponding to thesong and application to indicate that the user listened to the song andused the application. Moreover, social-networking system 860 may createa “played” edge 906 (as illustrated in FIG. 9) between concept nodes 904corresponding to the song and the application to indicate that theparticular song was played by the particular application. In this case,“played” edge 906 corresponds to an action performed by an externalapplication (e.g., a third-party online music application) on anexternal audio file (the song “Imagine”). Although this disclosuredescribes particular edges 906 with particular attributes connectinguser nodes 902 and concept nodes 904, this disclosure contemplates anysuitable edges 906 with any suitable attributes connecting user nodes902 and concept nodes 904. Moreover, although this disclosure describesedges between a user node 902 and a concept node 904 representing asingle relationship, this disclosure contemplates edges between a usernode 902 and a concept node 904 representing one or more relationships.As an example and not by way of limitation, an edge 906 may representboth that a user likes and has used at a particular concept.Alternatively, another edge 906 may represent each type of relationship(or multiples of a single relationship) between a user node 902 and aconcept node 904 (as illustrated in FIG. 9 between user node 902 foruser “E” and concept node 904 for “online music app”).

In particular embodiments, social-networking system 860 may create anedge 906 between a user node 902 and a concept node 904 in social graph900. As an example and not by way of limitation, a user viewing aconcept-profile page (such as, for example, by using a web browser or aspecial-purpose application hosted by the user's client system 830) mayindicate that he or she likes the concept represented by the conceptnode 904 by clicking or selecting a “Like” icon, which may cause theuser's client system 830 to send to social-networking system 860 amessage indicating the user's liking of the concept associated with theconcept-profile page. In response to the message, social-networkingsystem 860 may create an edge 906 between user node 902 associated withthe user and concept node 904, as illustrated by “like” edge 906 betweenthe user and concept node 904. In particular embodiments,social-networking system 860 may store an edge 906 in one or more datastores. In particular embodiments, an edge 906 may be automaticallyformed by social-networking system 860 in response to a particular useraction. As an example and not by way of limitation, if a first useruploads a picture, watches a movie, or listens to a song, an edge 906may be formed between user node 902 corresponding to the first user andconcept nodes 904 corresponding to those concepts. Although thisdisclosure describes forming particular edges 906 in particular manners,this disclosure contemplates forming any suitable edges 906 in anysuitable manner.

In particular embodiments, an advertisement may be text (which may beHTML-linked), one or more images (which may be HTML-linked), one or morevideos, audio, other suitable digital object files, a suitablecombination of these, or any other suitable advertisement in anysuitable digital format presented on one or more web pages, in one ormore e-mails, or in connection with search results requested by a user.In addition or as an alternative, an advertisement may be one or moresponsored stories (e.g., a news-feed or ticker item on social-networkingsystem 860). A sponsored story may be a social action by a user (such as“liking” a page, “liking” or commenting on a post on a page, RSVPing toan event associated with a page, voting on a question posted on a page,checking in to a place, using an application or playing a game, or“liking” or sharing a website) that an advertiser promotes, for example,by having the social action presented within a pre-determined area of aprofile page of a user or other page, presented with additionalinformation associated with the advertiser, bumped up or otherwisehighlighted within news feeds or tickers of other users, or otherwisepromoted. The advertiser may pay to have the social action promoted. Thesocial action may be promoted within or on social-networking system 860.In addition or as an alternative, the social action may be promotedoutside or off of social-networking system 860, where appropriate. Inparticular embodiments, a page may be an on-line presence (such as awebpage or website within or outside of social-networking system 860) ofa business, organization, or brand facilitating its sharing of storiesand connecting with people. A page may be customized, for example, byadding applications, posting stories, or hosting events.

A sponsored story may be generated from stories in users' news feeds andpromoted to specific areas within displays of users' web browsers whenviewing a web page associated with social-networking system 860.Sponsored stories are more likely to be viewed by users, at least inpart because sponsored stories generally involve interactions orsuggestions by the users' friends, fan pages, or other connections. Inconnection with sponsored stories, particular embodiments may utilizeone or more systems, components, elements, functions, methods,operations, or steps disclosed in U.S. patent application Ser. No.13/327,557, entitled “Sponsored Stories Unit Creation from OrganicActivity Stream” and filed 15 Dec. 2011, U.S. Patent ApplicationPublication No. 2012/0203831, entitled “Sponsored Stories Unit Creationfrom Organic Activity Stream” and filed 3 Feb. 2012 as U.S. patentapplication Ser. No. 13/020,745, or U.S. Patent Application PublicationNo. 2012/0233009, entitled “Endorsement Subscriptions for SponsoredStories” and filed 9 Mar. 2011 as U.S. patent application Ser. No.13/044,506, which are all incorporated herein by reference as an exampleand not by way of limitation. In particular embodiments, sponsoredstories may utilize computer-vision algorithms to detect products inuploaded images or photos lacking an explicit connection to anadvertiser as disclosed in U.S. patent application Ser. No. 13/212,356,entitled “Computer-Vision Content Detection for Sponsored Stories” andfiled 18 Aug. 2011, which is incorporated herein by reference as anexample and not by way of limitation.

As described above, an advertisement may be text (which may beHTML-linked), one or more images (which may be HTML-linked), one or morevideos, audio, one or more ADOBE FLASH files, a suitable combination ofthese, or any other suitable advertisement in any suitable digitalformat. In particular embodiments, an advertisement may be requested fordisplay within third-party webpages, social-networking-system webpages,or other pages. An advertisement may be displayed in a dedicated portionof a page, such as in a banner area at the top of the page, in a columnat the side of the page, in a GUI of the page, in a pop-up window, overthe top of content of the page, or elsewhere with respect to the page.In addition or as an alternative, an advertisement may be displayedwithin an application or within a game. An advertisement may bedisplayed within dedicated pages, requiring the user to interact with orwatch the advertisement before the user may access a page, utilize anapplication, or play a game. The user may, for example view theadvertisement through a web browser.

A user may interact with an advertisement in any suitable manner. Theuser may click or otherwise select the advertisement, and theadvertisement may direct the user (or a browser or other applicationbeing used by the user) to a page associated with the advertisement. Atthe page associated with the advertisement, the user may take additionalactions, such as purchasing a product or service associated with theadvertisement, receiving information associated with the advertisement,or subscribing to a newsletter associated with the advertisement. Anadvertisement with audio or video may be played by selecting a componentof the advertisement (like a “play button”). In particular embodiments,an advertisement may include one or more games, which a user or otherapplication may play in connection with the advertisement. Anadvertisement may include functionality for responding to a poll orquestion in the advertisement.

An advertisement may include social-networking-system functionality thata user may interact with. For example, an advertisement may enable auser to “like” or otherwise endorse the advertisement by selecting anicon or link associated with endorsement. Similarly, a user may sharethe advertisement with another user (e.g., through social-networkingsystem 860) or RSVP (e.g., through social-networking system 860) to anevent associated with the advertisement. In addition or as analternative, an advertisement may include social-networking-systemcontent directed to the user. For example, an advertisement may displayinformation about a friend of the user within social-networking system860 who has taken an action associated with the subject matter of theadvertisement.

Social-networking-system functionality or content may be associated withan advertisement in any suitable manner. For example, an advertisingsystem (which may include hardware, software, or both for receiving bidsfor advertisements and selecting advertisements in response) mayretrieve social-networking functionality or content fromsocial-networking system 860 and incorporate the retrievedsocial-networking functionality or content into the advertisement beforeserving the advertisement to a user. Examples of selecting and providingsocial-networking-system functionality or content with an advertisementare disclosed in U.S. Patent Application Publication No. 2012/0084160,entitled “Providing Social Endorsements with Online Advertising” andfiled 5 Oct. 2010 as U.S. patent application Ser. No. 12/898,662, and inU.S. Patent Application Publication No. 2012/0232998, entitled“Selecting Social Endorsement Information for an Advertisement forDisplay to a Viewing User” and filed 8 Mar. 2011 as U.S. patentapplication Ser. No. 13/043,424, which are both incorporated herein byreference as examples only and not by way of limitation. Interactingwith an advertisement that is associated with social-networking-systemfunctionality or content may cause information about the interaction tobe displayed in a profile page of the user in social-networking-system860.

Particular embodiments may facilitate the delivery of advertisements tousers that are more likely to find the advertisements more relevant oruseful. For example, an advertiser may realize higher conversion rates(and therefore higher return on investment (ROI) from advertising) byidentifying and targeting users that are more likely to find itsadvertisements more relevant or useful. The advertiser may useuser-profile information in social-networking system 860 to identifythose users. In addition or as an alternative, social-networking system860 may use user-profile information in social-networking system 860 toidentify those users for the advertiser. As examples and not by way oflimitation, particular embodiments may target users with the following:invitations or suggestions of events; suggestions regarding coupons,deals, or wish-list items; suggestions regarding friends' life events;suggestions regarding groups; advertisements; or social advertisements.Such targeting may occur, where appropriate, on or withinsocial-networking system 860, off or outside of social-networking system860, or on mobile computing devices of users. When on or withinsocial-networking system 860, such targeting may be directed to users'news feeds, search results, e-mail or other in-boxes, or notificationschannels or may appear in particular area of web pages ofsocial-networking system 860, such as a right-hand side of a web page ina concierge or grouper area (which may group along a right-hand railadvertisements associated with the same concept, node, or object) or anetwork-ego area (which may be based on what a user is viewing on theweb page and a current news feed of the user). When off or outside ofsocial-networking system 860, such targeting may be provided through athird-party website, e.g., involving an ad exchange or a social plug-in.When on a mobile computing device of a user, such targeting may beprovided through push notifications to the mobile computing device.

Targeting criteria used to identify and target users may includeexplicit, stated user interests on social-networking system 860 orexplicit connections of a user to a node, object, entity, brand, or pageon social-networking system 860. In addition or as an alternative, suchtargeting criteria may include implicit or inferred user interests orconnections (which may include analyzing a user's history, demographic,social or other activities, friends' social or other activities,subscriptions, or any of the preceding of other users similar to theuser (based, e.g., on shared interests, connections, or events)).Particular embodiments may utilize platform targeting, which may involveplatform and “like” impression data; contextual signals (e.g., “Who isviewing now or has viewed recently the page for [brand-name sodapop]?”); light-weight connections (e.g., “check-ins”); connectionlookalikes; fans; extracted keywords; EMU advertising; inferentialadvertising; coefficients, affinities, or other social-graphinformation; friends-of-friends connections; pinning or boosting; deals;polls; household income, social clusters or groups; products detected inimages or other media; social- or open-graph edge types; geo-prediction;views of profile or pages; status updates or other user posts (analysisof which may involve natural-language processing or keyword extraction);events information; or collaborative filtering. Identifying andtargeting users may also include privacy settings (such as useropt-outs), data hashing, or data anonymization, as appropriate.

To target users with advertisements, particular embodiments may utilizeone or more systems, components, elements, functions, methods,operations, or steps disclosed in the following, which are allincorporated herein by reference as examples and not by way oflimitation: U.S. Patent Application Publication No. 2009/0119167,entitled “Social Advertisements and Other Informational Messages on aSocial Networking Website and Advertising Model for Same” and filed 18Aug. 2008 as U.S. patent application Ser. No. 12/193,702; U.S. PatentApplication Publication No. 2009/0070219, entitled “TargetingAdvertisements in a Social Network” and filed 20 Aug. 2008 as U.S.patent application Ser. No. 12/195,321; U.S. Patent ApplicationPublication No. 2012/0158501, entitled “Targeting Social Advertising toFriends of Users Who Have Interacted With an Object Associated with theAdvertising” and filed 15 Dec. 2010 as U.S. patent application Ser. No.12/968,786; or U.S. Patent Application Publication No. 2012/0166532,entitled “Contextually Relevant Affinity Prediction in aSocial-Networking System” and filed 23 Dec. 2010 as U.S. patentapplication Ser. No. 12/978,265.

An advertisement may be presented or otherwise delivered using plug-insfor web browsers or other applications, iframe elements, news feeds,tickers, notifications (which may include, for example, e-mail, ShortMessage Service (SMS) messages, or notifications), or other means. Anadvertisement may be presented or otherwise delivered to a user on amobile or other computing device of the user. In connection withdelivering advertisements, particular embodiments may utilize one ormore systems, components, elements, functions, methods, operations, orsteps disclosed in the following, which are all incorporated herein byreference as examples and not by way of limitation: U.S. PatentApplication Publication No. 2012/0159635, entitled “Comment Plug-In forThird-Party System” and filed 15 Dec. 2010 as U.S. patent applicationSer. No. 12/969,368; U.S. Patent Application Publication No.2012/0158753, entitled “Comment Ordering System” and filed 15 Dec. 2010as U.S. patent application Ser. No. 12/969,408; U.S. Pat. No. 7,669,123,entitled “Dynamically Providing a News Feed About a User of a SocialNetwork” and filed 11 Aug. 2006 as U.S. patent application Ser. No.11/503,242; U.S. Pat. No. 8,402,094, entitled “Providing a NewsfeedBased on User Affinity for Entities and Monitored Actions in a SocialNetwork Environment” and filed 11 Aug. 2006 as U.S. patent applicationSer. No. 11/503,093; U.S. Patent Application Publication No.2012/0072428, entitled “Action Clustering for News Feeds” and filed 16Sep. 2010 as U.S. patent application Ser. No. 12/884,010; U.S. PatentApplication Publication No. 2011/0004692, entitled “GatheringInformation about Connections in a Social Networking Service” and filed1 Jul. 2009 as U.S. patent application Ser. No. 12/496,606; U.S. PatentApplication Publication No. 2008/0065701, entitled “Method and Systemfor Tracking Changes to User Content in an Online Social Network” andfiled 12 Sep. 2006 as U.S. patent application Ser. No. 11/531,154; U.S.Patent Application Publication No. 2008/0065604, entitled “FeedingUpdates to Landing Pages of Users of an Online Social Network fromExternal Sources” and filed 17 Jan. 2007 as U.S. patent application Ser.No. 11/624,088; U.S. Pat. No. 8,244,848, entitled “IntegratedSocial-Network Environment” and filed 19 Apr. 2010 as U.S. patentapplication Ser. No. 12/763,171; U.S. Patent Application Publication No.2011/0083101, entitled “Sharing of Location-Based Content Item inSocial-Networking Service” and filed 6 Oct. 2009 as U.S. patentapplication Ser. No. 12/574,614; U.S. Pat. No. 8,150,844, entitled“Location Ranking Using Social-Graph Information” and filed 18 Aug. 2010as U.S. patent application Ser. No. 12/858,718; U.S. patent applicationSer. No. 13/051,286, entitled “Sending Notifications to Users Based onUsers' Notification Tolerance Levels” and filed 18 Mar. 2011; U.S.patent application Ser. No. 13/096,184, entitled “Managing NotificationsPushed to User Devices” and filed 28 Apr. 2011; U.S. patent applicationSer. No. 13/276,248, entitled “Platform-Specific Notification DeliveryChannel” and filed 18 Oct. 2011; or U.S. Patent Application PublicationNo. 2012/0197709, entitled “Mobile Advertisement with Social Componentfor Geo-Social Networking System” and filed 1 Feb. 2011 as U.S. patentapplication Ser. No. 13/019,061. Although this disclosure describes orillustrates particular advertisements being delivered in particular waysand in connection with particular content, this disclosure contemplatesany suitable advertisements delivered in any suitable ways and inconnection with any suitable content.

In particular embodiments, social-networking system 860 may determinethe social-graph affinity (which may be referred to herein as“affinity”) of various social-graph entities for each other. Affinitymay represent the strength of a relationship or level of interestbetween particular objects associated with the online social network,such as users, concepts, content, actions, advertisements, other objectsassociated with the online social network, or any suitable combinationthereof. Affinity may also be determined with respect to objectsassociated with third-party systems 870 or other suitable systems. Anoverall affinity for a social-graph entity for each user, subjectmatter, or type of content may be established. The overall affinity maychange based on continued monitoring of the actions or relationshipsassociated with the social-graph entity. Although this disclosuredescribes determining particular affinities in a particular manner, thisdisclosure contemplates determining any suitable affinities in anysuitable manner.

In particular embodiments, social-networking system 860 may measure orquantify social-graph affinity using an affinity coefficient (which maybe referred to herein as “coefficient”). The coefficient may representor quantify the strength of a relationship between particular objectsassociated with the online social network. The coefficient may alsorepresent a probability or function that measures a predictedprobability that a user will perform a particular action based on theuser's interest in the action. In this way, a user's future actions maybe predicted based on the user's prior actions, where the coefficientmay be calculated at least in part on the history of the user's actions.Coefficients may be used to predict any number of actions, which may bewithin or outside of the online social network. As an example and not byway of limitation, these actions may include various types ofcommunications, such as sending messages, posting content, or commentingon content; various types of observation actions, such as accessing orviewing profile pages, media, or other suitable content; various typesof coincidence information about two or more social-graph entities, suchas being in the same group, tagged in the same photograph, checked-in atthe same location, or attending the same event; or other suitableactions. Although this disclosure describes measuring affinity in aparticular manner, this disclosure contemplates measuring affinity inany suitable manner.

In particular embodiments, social-networking system 860 may use avariety of factors to calculate a coefficient. These factors mayinclude, for example, user actions, types of relationships betweenobjects, location information, other suitable factors, or anycombination thereof. In particular embodiments, different factors may beweighted differently when calculating the coefficient. The weights foreach factor may be static or the weights may change according to, forexample, the user, the type of relationship, the type of action, theuser's location, and so forth. Ratings for the factors may be combinedaccording to their weights to determine an overall coefficient for theuser. As an example and not by way of limitation, particular useractions may be assigned both a rating and a weight while a relationshipassociated with the particular user action is assigned a rating and acorrelating weight (e.g., so the weights total 100%). To calculate thecoefficient of a user towards a particular object, the rating assignedto the user's actions may comprise, for example, 60% of the overallcoefficient, while the relationship between the user and the object maycomprise 40% of the overall coefficient. In particular embodiments, thesocial-networking system 860 may consider a variety of variables whendetermining weights for various factors used to calculate a coefficient,such as, for example, the time since information was accessed, decayfactors, frequency of access, relationship to information orrelationship to the object about which information was accessed,relationship to social-graph entities connected to the object, short- orlong-term averages of user actions, user feedback, other suitablevariables, or any combination thereof. As an example and not by way oflimitation, a coefficient may include a decay factor that causes thestrength of the signal provided by particular actions to decay withtime, such that more recent actions are more relevant when calculatingthe coefficient. The ratings and weights may be continuously updatedbased on continued tracking of the actions upon which the coefficient isbased. Any type of process or algorithm may be employed for assigning,combining, averaging, and so forth the ratings for each factor and theweights assigned to the factors. In particular embodiments,social-networking system 860 may determine coefficients usingmachine-learning algorithms trained on historical actions and past userresponses, or data farmed from users by exposing them to various optionsand measuring responses. Although this disclosure describes calculatingcoefficients in a particular manner, this disclosure contemplatescalculating coefficients in any suitable manner.

In particular embodiments, social-networking system 860 may calculate acoefficient based on a user's actions. Social-networking system 860 maymonitor such actions on the online social network, on a third-partysystem 870, on other suitable systems, or any combination thereof. Anysuitable type of user actions may be tracked or monitored. Typical useractions include viewing profile pages, creating or posting content,interacting with content, tagging or being tagged in images, joininggroups, listing and confirming attendance at events, checking-in atlocations, liking particular pages, creating pages, and performing othertasks that facilitate social action. In particular embodiments,social-networking system 860 may calculate a coefficient based on theuser's actions with particular types of content. The content may beassociated with the online social network, a third-party system 870, oranother suitable system. The content may include users, profile pages,posts, news stories, headlines, instant messages, chat roomconversations, emails, advertisements, pictures, video, music, othersuitable objects, or any combination thereof. Social-networking system860 may analyze a user's actions to determine whether one or more of theactions indicate an affinity for subject matter, content, other users,and so forth. As an example and not by way of limitation, if a userfrequently posts content related to “coffee” or variants thereof,social-networking system 860 may determine the user has a highcoefficient with respect to the concept “coffee”. Particular actions ortypes of actions may be assigned a higher weight and/or rating thanother actions, which may affect the overall calculated coefficient. Asan example and not by way of limitation, if a first user emails a seconduser, the weight or the rating for the action may be higher than if thefirst user simply views the user-profile page for the second user.

In particular embodiments, social-networking system 860 may calculate acoefficient based on the type of relationship between particularobjects. Referencing the social graph 900, social-networking system 860may analyze the number and/or type of edges 906 connecting particularuser nodes 902 and concept nodes 904 when calculating a coefficient. Asan example and not by way of limitation, user nodes 902 that areconnected by a spouse-type edge (representing that the two users aremarried) may be assigned a higher coefficient than a user nodes 902 thatare connected by a friend-type edge. In other words, depending upon theweights assigned to the actions and relationships for the particularuser, the overall affinity may be determined to be higher for contentabout the user's spouse than for content about the user's friend. Inparticular embodiments, the relationships a user has with another objectmay affect the weights and/or the ratings of the user's actions withrespect to calculating the coefficient for that object. As an exampleand not by way of limitation, if a user is tagged in a first photo, butmerely likes a second photo, social-networking system 860 may determinethat the user has a higher coefficient with respect to the first photothan the second photo because having a tagged-in-type relationship withcontent may be assigned a higher weight and/or rating than having alike-type relationship with content. In particular embodiments,social-networking system 860 may calculate a coefficient for a firstuser based on the relationship one or more second users have with aparticular object. In other words, the connections and coefficientsother users have with an object may affect the first user's coefficientfor the object. As an example and not by way of limitation, if a firstuser is connected to or has a high coefficient for one or more secondusers, and those second users are connected to or have a highcoefficient for a particular object, social-networking system 860 maydetermine that the first user should also have a relatively highcoefficient for the particular object. In particular embodiments, thecoefficient may be based on the degree of separation between particularobjects. The lower coefficient may represent the decreasing likelihoodthat the first user will share an interest in content objects of theuser that is indirectly connected to the first user in the social graph900. As an example and not by way of limitation, social-graph entitiesthat are closer in the social graph 900 (i.e., fewer degrees ofseparation) may have a higher coefficient than entities that are furtherapart in the social graph 900.

In particular embodiments, social-networking system 860 may calculate acoefficient based on location information. Objects that aregeographically closer to each other may be considered to be more relatedor of more interest to each other than more distant objects. Inparticular embodiments, the coefficient of a user towards a particularobject may be based on the proximity of the object's location to acurrent location associated with the user (or the location of a clientsystem 830 of the user). A first user may be more interested in otherusers or concepts that are closer to the first user. As an example andnot by way of limitation, if a user is one mile from an airport and twomiles from a gas station, social-networking system 860 may determinethat the user has a higher coefficient for the airport than the gasstation based on the proximity of the airport to the user.

In particular embodiments, social-networking system 860 may performparticular actions with respect to a user based on coefficientinformation. Coefficients may be used to predict whether a user willperform a particular action based on the user's interest in the action.A coefficient may be used when generating or presenting any type ofobjects to a user, such as advertisements, search results, news stories,media, messages, notifications, or other suitable objects. Thecoefficient may also be utilized to rank and order such objects, asappropriate. In this way, social-networking system 860 may provideinformation that is relevant to user's interests and currentcircumstances, increasing the likelihood that they will find suchinformation of interest. In particular embodiments, social-networkingsystem 860 may generate content based on coefficient information.Content objects may be provided or selected based on coefficientsspecific to a user. As an example and not by way of limitation, thecoefficient may be used to generate media for the user, where the usermay be presented with media for which the user has a high overallcoefficient with respect to the media object. As another example and notby way of limitation, the coefficient may be used to generateadvertisements for the user, where the user may be presented withadvertisements for which the user has a high overall coefficient withrespect to the advertised object. In particular embodiments,social-networking system 860 may generate search results based oncoefficient information. Search results for a particular user may bescored or ranked based on the coefficient associated with the searchresults with respect to the querying user. As an example and not by wayof limitation, search results corresponding to objects with highercoefficients may be ranked higher on a search-results page than resultscorresponding to objects having lower coefficients.

In particular embodiments, social-networking system 860 may calculate acoefficient in response to a request for a coefficient from a particularsystem or process. To predict the likely actions a user may take (or maybe the subject of) in a given situation, any process may request acalculated coefficient for a user. The request may also include a set ofweights to use for various factors used to calculate the coefficient.This request may come from a process running on the online socialnetwork, from a third-party system 870 (e.g., via an API or othercommunication channel), or from another suitable system. In response tothe request, social-networking system 860 may calculate the coefficient(or access the coefficient information if it has previously beencalculated and stored). In particular embodiments, social-networkingsystem 860 may measure an affinity with respect to a particular process.Different processes (both internal and external to the online socialnetwork) may request a coefficient for a particular object or set ofobjects. Social-networking system 860 may provide a measure of affinitythat is relevant to the particular process that requested the measure ofaffinity. In this way, each process receives a measure of affinity thatis tailored for the different context in which the process will use themeasure of affinity.

In connection with social-graph affinity and affinity coefficients,particular embodiments may utilize one or more systems, components,elements, functions, methods, operations, or steps disclosed in U.S.patent application Ser. No. 11/503,093, filed 11 Aug. 2006, U.S. patentapplication Ser. No. 12/977,027, filed 22 Dec. 2010, U.S. patentapplication Ser. No. 12/978,265, filed 23 Dec. 2010, and U.S. patentapplication Ser. No. 13/632,869, filed 1 Oct. 2012, each of which isincorporated by reference.

In particular embodiments, one or more of the content objects of theonline social network may be associated with a privacy setting. Theprivacy settings (or “access settings”) for an object may be stored inany suitable manner, such as, for example, in association with theobject, in an index on an authorization server, in another suitablemanner, or any combination thereof. A privacy setting of an object mayspecify how the object (or particular information associated with anobject) can be accessed (e.g., viewed or shared) using the online socialnetwork. Where the privacy settings for an object allow a particularuser to access that object, the object may be described as being“visible” with respect to that user. As an example and not by way oflimitation, a user of the online social network may specify privacysettings for a user-profile page that identify a set of users that mayaccess the work experience information on the user-profile page, thusexcluding other users from accessing the information. In particularembodiments, the privacy settings may specify a “blocked list” of usersthat should not be allowed to access certain information associated withthe object. In other words, the blocked list may specify one or moreusers or entities for which an object is not visible. As an example andnot by way of limitation, a user may specify a set of users that may notaccess photos albums associated with the user, thus excluding thoseusers from accessing the photo albums (while also possibly allowingcertain users not within the set of users to access the photo albums).In particular embodiments, privacy settings may be associated withparticular social-graph elements. Privacy settings of a social-graphelement, such as a node or an edge, may specify how the social-graphelement, information associated with the social-graph element, orcontent objects associated with the social-graph element can be accessedusing the online social network. As an example and not by way oflimitation, a particular concept node 204 corresponding to a particularphoto may have a privacy setting specifying that the photo may only beaccessed by users tagged in the photo and their friends. In particularembodiments, privacy settings may allow users to opt in or opt out ofhaving their actions logged by social-networking system 860 or sharedwith other systems (e.g., third-party system 870). In particularembodiments, the privacy settings associated with an object may specifyany suitable granularity of permitted access or denial of access. As anexample and not by way of limitation, access or denial of access may bespecified for particular users (e.g., only me, my roommates, and myboss), users within a particular degrees-of-separation (e.g., friends,or friends-of-friends), user groups (e.g., the gaming club, my family),user networks (e.g., employees of particular employers, students oralumni of particular university), all users (“public”), no users(“private”), users of third-party systems 870, particular applications(e.g., third-party applications, external websites), other suitableusers or entities, or any combination thereof. Although this disclosuredescribes using particular privacy settings in a particular manner, thisdisclosure contemplates using any suitable privacy settings in anysuitable manner.

In particular embodiments, one or more servers 862 may beauthorization/privacy servers for enforcing privacy settings. Inresponse to a request from a user (or other entity) for a particularobject stored in a data store 864, social-networking system 860 may senda request to the data store 864 for the object. The request may identifythe user associated with the request and may only be sent to the user(or a client system 830 of the user) if the authorization serverdetermines that the user is authorized to access the object based on theprivacy settings associated with the object. If the requesting user isnot authorized to access the object, the authorization server mayprevent the requested object from being retrieved from the data store864, or may prevent the requested object from being sent to the user. Inthe search query context, an object may only be generated as a searchresult if the querying user is authorized to access the object. In otherwords, the object must have a visibility that is visible to the queryinguser. If the object has a visibility that is not visible to the user,the object may be excluded from the search results. Although thisdisclosure describes enforcing privacy settings in a particular manner,this disclosure contemplates enforcing privacy settings in any suitablemanner.

FIG. 10 illustrates an example computer system 1000. In particularembodiments, one or more computer systems 1000 perform one or more stepsof one or more methods described or illustrated herein. In particularembodiments, one or more computer systems 1000 provide functionalitydescribed or illustrated herein. In particular embodiments, softwarerunning on one or more computer systems 1000 performs one or more stepsof one or more methods described or illustrated herein or providesfunctionality described or illustrated herein. Particular embodimentsinclude one or more portions of one or more computer systems 1000.Herein, reference to a computer system may encompass a computing device,and vice versa, where appropriate. Moreover, reference to a computersystem may encompass one or more computer systems, where appropriate.

This disclosure contemplates any suitable number of computer systems1000. This disclosure contemplates computer system 1000 taking anysuitable physical form. As example and not by way of limitation,computer system 1000 may be an embedded computer system, asystem-on-chip (SOC), a single-board computer system (SBC) (such as, forexample, a computer-on-module (COM) or system-on-module (SOM)), adesktop computer system, a laptop or notebook computer system, aninteractive kiosk, a mainframe, a mesh of computer systems, a mobiletelephone, a personal digital assistant (PDA), a server, a tabletcomputer system, an augmented/virtual reality device, or a combinationof two or more of these. Where appropriate, computer system 1000 mayinclude one or more computer systems 1000; be unitary or distributed;span multiple locations; span multiple machines; span multiple datacenters; or reside in a cloud, which may include one or more cloudcomponents in one or more networks. Where appropriate, one or morecomputer systems 1000 may perform without substantial spatial ortemporal limitation one or more steps of one or more methods describedor illustrated herein. As an example and not by way of limitation, oneor more computer systems 1000 may perform in real time or in batch modeone or more steps of one or more methods described or illustratedherein. One or more computer systems 1000 may perform at different timesor at different locations one or more steps of one or more methodsdescribed or illustrated herein, where appropriate.

In particular embodiments, computer system 1000 includes a processor1002, memory 1004, storage 1006, an input/output (I/O) interface 1008, acommunication interface 1010, and a bus 1012. Although this disclosuredescribes and illustrates a particular computer system having aparticular number of particular components in a particular arrangement,this disclosure contemplates any suitable computer system having anysuitable number of any suitable components in any suitable arrangement.

In particular embodiments, processor 1002 includes hardware forexecuting instructions, such as those making up a computer program. Asan example and not by way of limitation, to execute instructions,processor 1002 may retrieve (or fetch) the instructions from an internalregister, an internal cache, memory 1004, or storage 1006; decode andexecute them; and then write one or more results to an internalregister, an internal cache, memory 1004, or storage 1006. In particularembodiments, processor 1002 may include one or more internal caches fordata, instructions, or addresses. This disclosure contemplates processor1002 including any suitable number of any suitable internal caches,where appropriate. As an example and not by way of limitation, processor1002 may include one or more instruction caches, one or more datacaches, and one or more translation lookaside buffers (TLBs).Instructions in the instruction caches may be copies of instructions inmemory 1004 or storage 1006, and the instruction caches may speed upretrieval of those instructions by processor 1002. Data in the datacaches may be copies of data in memory 1004 or storage 1006 forinstructions executing at processor 1002 to operate on; the results ofprevious instructions executed at processor 1002 for access bysubsequent instructions executing at processor 1002 or for writing tomemory 1004 or storage 1006; or other suitable data. The data caches mayspeed up read or write operations by processor 1002. The TLBs may speedup virtual-address translation for processor 1002. In particularembodiments, processor 1002 may include one or more internal registersfor data, instructions, or addresses. This disclosure contemplatesprocessor 1002 including any suitable number of any suitable internalregisters, where appropriate. Where appropriate, processor 1002 mayinclude one or more arithmetic logic units (ALUs); be a multi-coreprocessor; or include one or more processors 1002. Although thisdisclosure describes and illustrates a particular processor, thisdisclosure contemplates any suitable processor.

In particular embodiments, memory 1004 includes main memory for storinginstructions for processor 1002 to execute or data for processor 1002 tooperate on. As an example and not by way of limitation, computer system1000 may load instructions from storage 1006 or another source (such as,for example, another computer system 1000) to memory 1004. Processor1002 may then load the instructions from memory 1004 to an internalregister or internal cache. To execute the instructions, processor 1002may retrieve the instructions from the internal register or internalcache and decode them. During or after execution of the instructions,processor 1002 may write one or more results (which may be intermediateor final results) to the internal register or internal cache. Processor1002 may then write one or more of those results to memory 1004. Inparticular embodiments, processor 1002 executes only instructions in oneor more internal registers or internal caches or in memory 1004 (asopposed to storage 1006 or elsewhere) and operates only on data in oneor more internal registers or internal caches or in memory 1004 (asopposed to storage 1006 or elsewhere). One or more memory buses (whichmay each include an address bus and a data bus) may couple processor1002 to memory 1004. Bus 1012 may include one or more memory buses, asdescribed below. In particular embodiments, one or more memorymanagement units (MMUs) reside between processor 1002 and memory 1004and facilitate accesses to memory 1004 requested by processor 1002. Inparticular embodiments, memory 1004 includes random access memory (RAM).This RAM may be volatile memory, where appropriate Where appropriate,this RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Moreover, whereappropriate, this RAM may be single-ported or multi-ported RAM. Thisdisclosure contemplates any suitable RAM. Memory 1004 may include one ormore memories 1004, where appropriate. Although this disclosuredescribes and illustrates particular memory, this disclosurecontemplates any suitable memory.

In particular embodiments, storage 1006 includes mass storage for dataor instructions. As an example and not by way of limitation, storage1006 may include a hard disk drive (HDD), a floppy disk drive, flashmemory, an optical disc, a magneto-optical disc, magnetic tape, or aUniversal Serial Bus (USB) drive or a combination of two or more ofthese. Storage 1006 may include removable or non-removable (or fixed)media, where appropriate. Storage 1006 may be internal or external tocomputer system 1000, where appropriate. In particular embodiments,storage 1006 is non-volatile, solid-state memory. In particularembodiments, storage 1006 includes read-only memory (ROM). Whereappropriate, this ROM may be mask-programmed ROM, programmable ROM(PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM),electrically alterable ROM (EAROM), or flash memory or a combination oftwo or more of these. This disclosure contemplates mass storage 1006taking any suitable physical form. Storage 1006 may include one or morestorage control units facilitating communication between processor 1002and storage 1006, where appropriate. Where appropriate, storage 1006 mayinclude one or more storages 1006. Although this disclosure describesand illustrates particular storage, this disclosure contemplates anysuitable storage.

In particular embodiments, I/O interface 1008 includes hardware,software, or both, providing one or more interfaces for communicationbetween computer system 1000 and one or more I/O devices. Computersystem 1000 may include one or more of these I/O devices, whereappropriate. One or more of these I/O devices may enable communicationbetween a person and computer system 1000. As an example and not by wayof limitation, an I/O device may include a keyboard, keypad, microphone,monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet,touch screen, trackball, video camera, another suitable I/O device or acombination of two or more of these. An I/O device may include one ormore sensors. This disclosure contemplates any suitable I/O devices andany suitable I/O interfaces 1008 for them. Where appropriate, I/Ointerface 1008 may include one or more device or software driversenabling processor 1002 to drive one or more of these I/O devices. I/Ointerface 1008 may include one or more I/O interfaces 1008, whereappropriate. Although this disclosure describes and illustrates aparticular I/O interface, this disclosure contemplates any suitable I/Ointerface.

In particular embodiments, communication interface 1010 includeshardware, software, or both providing one or more interfaces forcommunication (such as, for example, packet-based communication) betweencomputer system 1000 and one or more other computer systems 1000 or oneor more networks. As an example and not by way of limitation,communication interface 1010 may include a network interface controller(NIC) or network adapter for communicating with an Ethernet or otherwire-based network or a wireless NIC (WNIC) or wireless adapter forcommunicating with a wireless network, such as a WI-FI network. Thisdisclosure contemplates any suitable network and any suitablecommunication interface 1010 for it. As an example and not by way oflimitation, computer system 1000 may communicate with an ad hoc network,a personal area network (PAN), a local area network (LAN), a wide areanetwork (WAN), a metropolitan area network (MAN), or one or moreportions of the Internet or a combination of two or more of these. Oneor more portions of one or more of these networks may be wired orwireless. As an example, computer system 1000 may communicate with awireless PAN (WPAN) (such as, for example, a BLUETOOTH WPAN), a WI-FInetwork, a WI-MAX network, a cellular telephone network (such as, forexample, a Global System for Mobile Communications (GSM) network), orother suitable wireless network or a combination of two or more ofthese. Computer system 1000 may include any suitable communicationinterface 1010 for any of these networks, where appropriate.Communication interface 1010 may include one or more communicationinterfaces 1010, where appropriate. Although this disclosure describesand illustrates a particular communication interface, this disclosurecontemplates any suitable communication interface.

In particular embodiments, bus 1012 includes hardware, software, or bothcoupling components of computer system 1000 to each other. As an exampleand not by way of limitation, bus 1012 may include an AcceleratedGraphics Port (AGP) or other graphics bus, an Enhanced Industry StandardArchitecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT)interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBANDinterconnect, a low-pin-count (LPC) bus, a memory bus, a Micro ChannelArchitecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, aPCI-Express (PCIe) bus, a serial advanced technology attachment (SATA)bus, a Video Electronics Standards Association local (VLB) bus, oranother suitable bus or a combination of two or more of these. Bus 1012may include one or more buses 1012, where appropriate. Although thisdisclosure describes and illustrates a particular bus, this disclosurecontemplates any suitable bus or interconnect.

Herein, a computer-readable non-transitory storage medium or media mayinclude one or more semiconductor-based or other integrated circuits(ICs) (such, as for example, field-programmable gate arrays (FPGAs) orapplication-specific ICs (ASICs)), hard disk drives (HDDs), hybrid harddrives (HHDs), optical discs, optical disc drives (ODDs),magneto-optical discs, magneto-optical drives, floppy diskettes, floppydisk drives (FDDs), magnetic tapes, solid-state drives (SSDs),RAM-drives, SECURE DIGITAL cards or drives, any other suitablecomputer-readable non-transitory storage media, or any suitablecombination of two or more of these, where appropriate. Acomputer-readable non-transitory storage medium may be volatile,non-volatile, or a combination of volatile and non-volatile, whereappropriate.

Herein, “or” is inclusive and not exclusive, unless expressly indicatedotherwise or indicated otherwise by context. Therefore, herein, “A or B”means “A, B, or both,” unless expressly indicated otherwise or indicatedotherwise by context. Moreover, “and” is both joint and several, unlessexpressly indicated otherwise or indicated otherwise by context.Therefore, herein, “A and B” means “A and B, jointly or severally,”unless expressly indicated otherwise or indicated otherwise by context.

The scope of this disclosure encompasses all changes, substitutions,variations, alterations, and modifications to the example embodimentsdescribed or illustrated herein that a person having ordinary skill inthe art would comprehend. The scope of this disclosure is not limited tothe example embodiments described or illustrated herein. Moreover,although this disclosure describes and illustrates respectiveembodiments herein as including particular components, elements,feature, functions, operations, or steps, any of these embodiments mayinclude any combination or permutation of any of the components,elements, features, functions, operations, or steps described orillustrated anywhere herein that a person having ordinary skill in theart would comprehend. Furthermore, reference in the appended claims toan apparatus or system or a component of an apparatus or system beingadapted to, arranged to, capable of, configured to, enabled to, operableto, or operative to perform a particular function encompasses thatapparatus, system, component, whether or not it or that particularfunction is activated, turned on, or unlocked, as long as thatapparatus, system, or component is so adapted, arranged, capable,configured, enabled, operable, or operative. Additionally, although thisdisclosure describes or illustrates particular embodiments as providingparticular advantages, particular embodiments may provide none, some, orall of these advantages.

What is claimed is:
 1. A method comprising, by one or more computingdevices: receiving a plurality of input frames of a video sequenceassociated with a time t; predicting, using a convolutional network, oneor more future frames of the video sequence from the plurality of inputframes, wherein the convolutional network is trained with randomlyselected temporal sequences of a n×m grid of pixels from the pluralityof input frames exhibiting a threshold of optical flow; and outputting afirst future frame of the video sequence associated with a time t+1 aspredicted by the convolutional network.
 2. The method of claim 1,further comprising by the one or more computing devices: predicting,using the convolutional network, a second future frame of the videosequence associated with time t+2 by using the first future frame as aninput frame of the video sequence; recursively applying theconvolutional network based on the first future frame; and outputtingthe second future frame of the video sequence associated with the timet+2 as predicted by the convolutional network.
 3. The method of claim 1,wherein the threshold of optical flow is determined based on acomparison of movement in the n×m grid of pixels between a time t−1 andthe time t.
 4. The method of claim 3, wherein the comparison of movementcomprises comparing pixel intensity associated with each pixel of then×m grid of pixels between the time t−1 and the time t and calculating asimilarity measure based on compared pixel intensities.
 5. The method ofclaim 1, wherein the convolutional network is further trained byevaluating a quality of the predicted one or more future frames of thevideo sequence by calculating a peak-signal-to-noise ratio between theone or more future frames of the video sequence and a ground truth frameof the video sequence.
 6. The method of claim 1, wherein theconvolutional network is further trained by evaluating a quality of thepredicted one or more future frames of the video sequence by calculatinga structural similarity index measure between the one or more futureframes of the video sequence and a ground truth frame of the videosequence.
 7. The method of claim 1, wherein the convolutional networkemploys an L₁ loss to train the network on the input frames.
 8. Themethod of claim 1, wherein the convolutional network employs an L₂ lossto train the network on the input frames.
 9. The method of claim 1,wherein the convolutional network employs a gradient difference loss totrain the network on the input frames.
 10. One or more computer-readablenon-transitory storage media embodying software that is operable whenexecuted to: receive a plurality of input frames of a video sequenceassociated with a time t; predict, using a convolutional network, one ormore future frames of the video sequence from the plurality of inputframes, wherein the convolutional network is trained with randomlyselected temporal sequences of a n×m grid of pixels from the pluralityof input frames exhibiting a threshold of optical flow; and output afirst future frame of the video sequence associated with a time t+1 aspredicted by the convolutional network.
 11. The media of claim 10,wherein threshold of optical flow is determined based on a comparison ofmovement in the n×m grid of pixels between a time t−1 and the time t.12. The media of claim 11, wherein the comparison of movement comprisescomparing pixel intensity associated with each pixel of the n×m grid ofpixels between the time t−1 and the time t and calculating a similaritymeasure based on compared pixel intensities.
 13. The media of claim 10,wherein the training of the convolutional network may further compriseevaluating a quality of the predicted one or more future frames of thevideo sequence by calculating a peak-signal-to-noise ratio between theone or more future frames of the video sequence and a ground truth frameof the video sequence.
 14. The media of claim 10, wherein the trainingof the convolutional network may further comprise evaluating a qualityof the predicted one or more future frames of the video sequence bycalculating a structural similarity index measure between the one ormore future frames of the video sequence and a ground truth frame of thevideo sequence.
 15. A system comprising: one or more processors; and oneor more computer-readable non-transitory storage media coupled to one ormore of the processors and comprising instructions operable whenexecuted by one or more of the processors to cause the system to:receive a plurality of input frames of a video sequence associated witha time t; predict, using a convolutional network, one or more futureframes of the video sequence from the plurality of input frames, whereinthe convolutional network is trained with randomly selected temporalsequences of a n×m grid of pixels from the plurality of input framesexhibiting a threshold of optical flow; and output a first future frameof the video sequence associated with a time t+1 as predicted by theconvolutional network.
 16. The system of claim 15, wherein theprocessors are further operable when executing the instructions to:predict a second future frame of the video sequence associated with timet+2 by using the first future frame as an input frame of the videosequence, recursively apply the generative model based on the firstfuture frame, and outputting the second future frame of the videosequence associated with the time t+2 as predicted by the generativemodel.
 17. The system of claim 15, wherein threshold of optical flow isdetermined based on a comparison of movement in the n×m grid of pixelsbetween a time t−1 and the time t.
 18. The system of claim 17, whereinthe comparison of movement comprises comparing pixel intensityassociated with each pixel of the n×m grid of pixels between the timet−1 and the time t and calculating a similarity measure based oncompared pixel intensities.
 19. The system of claim 15, wherein thetraining of the convolutional network may further comprise evaluating aquality of the predicted one or more future frames of the video sequenceby calculating a peak-signal-to-noise ratio between the one or morefuture frames of the video sequence and a ground truth frame of thevideo sequence.
 20. The system of claim 15, wherein the training of theconvolutional network may further comprise evaluating a quality of thepredicted one or more future frames of the video sequence by calculatinga structural similarity index measure between the one or more futureframes of the video sequence and a ground truth frame of the videosequence.